2023-02-13 21:57:26,412 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822 2023-02-13 21:57:26,424 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClientSimple timeout: 13 mins 2023-02-13 21:57:26,437 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-13 21:57:26,437 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/cluster_c8eab63c-2879-73ac-9f0f-8254d034d5a9, deleteOnExit=true 2023-02-13 21:57:26,438 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-13 21:57:26,438 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/test.cache.data in system properties and HBase conf 2023-02-13 21:57:26,439 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/hadoop.tmp.dir in system properties and HBase conf 2023-02-13 21:57:26,439 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/hadoop.log.dir in system properties and HBase conf 2023-02-13 21:57:26,440 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-13 21:57:26,440 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-13 21:57:26,440 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-13 21:57:26,555 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-02-13 21:57:26,927 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-13 21:57:26,930 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-13 21:57:26,931 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-13 21:57:26,931 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-13 21:57:26,931 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-13 21:57:26,932 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-13 21:57:26,932 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-13 21:57:26,932 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-13 21:57:26,933 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-13 21:57:26,933 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-13 21:57:26,934 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/nfs.dump.dir in system properties and HBase conf 2023-02-13 21:57:26,934 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/java.io.tmpdir in system properties and HBase conf 2023-02-13 21:57:26,934 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-13 21:57:26,935 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-13 21:57:26,935 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-13 21:57:27,409 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-13 21:57:27,413 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-13 21:57:27,665 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-02-13 21:57:27,815 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-02-13 21:57:27,831 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 21:57:27,873 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 21:57:27,911 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/java.io.tmpdir/Jetty_localhost_localdomain_33885_hdfs____.mzrrl/webapp 2023-02-13 21:57:28,026 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:33885 2023-02-13 21:57:28,054 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-13 21:57:28,055 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-13 21:57:28,406 WARN [Listener at localhost.localdomain/37775] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 21:57:28,483 WARN [Listener at localhost.localdomain/37775] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 21:57:28,497 WARN [Listener at localhost.localdomain/37775] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 21:57:28,503 INFO [Listener at localhost.localdomain/37775] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 21:57:28,507 INFO [Listener at localhost.localdomain/37775] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/java.io.tmpdir/Jetty_localhost_44147_datanode____7x2zff/webapp 2023-02-13 21:57:28,612 INFO [Listener at localhost.localdomain/37775] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44147 2023-02-13 21:57:28,929 WARN [Listener at localhost.localdomain/41023] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 21:57:28,938 WARN [Listener at localhost.localdomain/41023] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 21:57:28,941 WARN [Listener at localhost.localdomain/41023] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 21:57:28,943 INFO [Listener at localhost.localdomain/41023] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 21:57:28,949 INFO [Listener at localhost.localdomain/41023] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/java.io.tmpdir/Jetty_localhost_38231_datanode____e66lf6/webapp 2023-02-13 21:57:29,032 INFO [Listener at localhost.localdomain/41023] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38231 2023-02-13 21:57:29,040 WARN [Listener at localhost.localdomain/36215] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 21:57:29,052 WARN [Listener at localhost.localdomain/36215] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 21:57:29,055 WARN [Listener at localhost.localdomain/36215] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 21:57:29,056 INFO [Listener at localhost.localdomain/36215] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 21:57:29,061 INFO [Listener at localhost.localdomain/36215] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/java.io.tmpdir/Jetty_localhost_42645_datanode____h3xjds/webapp 2023-02-13 21:57:29,151 INFO [Listener at localhost.localdomain/36215] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42645 2023-02-13 21:57:29,158 WARN [Listener at localhost.localdomain/44075] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 21:57:29,335 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x36dcc94d85a42e6e: Processing first storage report for DS-1ba6ad40-f2ea-4538-8465-3514bd0a9251 from datanode 20ff377b-2559-4f6e-af23-080f5371a3dc 2023-02-13 21:57:29,336 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x36dcc94d85a42e6e: from storage DS-1ba6ad40-f2ea-4538-8465-3514bd0a9251 node DatanodeRegistration(127.0.0.1:36475, datanodeUuid=20ff377b-2559-4f6e-af23-080f5371a3dc, infoPort=43225, infoSecurePort=0, ipcPort=36215, storageInfo=lv=-57;cid=testClusterID;nsid=117908355;c=1676325447478), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-02-13 21:57:29,337 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xf9a3b379be600f37: Processing first storage report for DS-951ea18f-1dbb-4a4d-8960-2e8bc8a798dd from datanode 26a0f3f3-fab9-4883-9582-fbed1359d57e 2023-02-13 21:57:29,337 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xf9a3b379be600f37: from storage DS-951ea18f-1dbb-4a4d-8960-2e8bc8a798dd node DatanodeRegistration(127.0.0.1:40923, datanodeUuid=26a0f3f3-fab9-4883-9582-fbed1359d57e, infoPort=33295, infoSecurePort=0, ipcPort=44075, storageInfo=lv=-57;cid=testClusterID;nsid=117908355;c=1676325447478), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 21:57:29,337 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x9b20fb7efe69b521: Processing first storage report for DS-0adfe1ee-30f2-40dc-88b6-d68d1d5073c7 from datanode 0a762196-68ef-4608-a8ad-5faf41155fc5 2023-02-13 21:57:29,337 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x9b20fb7efe69b521: from storage DS-0adfe1ee-30f2-40dc-88b6-d68d1d5073c7 node DatanodeRegistration(127.0.0.1:34727, datanodeUuid=0a762196-68ef-4608-a8ad-5faf41155fc5, infoPort=35465, infoSecurePort=0, ipcPort=41023, storageInfo=lv=-57;cid=testClusterID;nsid=117908355;c=1676325447478), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 21:57:29,337 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x36dcc94d85a42e6e: Processing first storage report for DS-18615db0-e795-4b14-bc28-678e01131a21 from datanode 20ff377b-2559-4f6e-af23-080f5371a3dc 2023-02-13 21:57:29,337 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x36dcc94d85a42e6e: from storage DS-18615db0-e795-4b14-bc28-678e01131a21 node DatanodeRegistration(127.0.0.1:36475, datanodeUuid=20ff377b-2559-4f6e-af23-080f5371a3dc, infoPort=43225, infoSecurePort=0, ipcPort=36215, storageInfo=lv=-57;cid=testClusterID;nsid=117908355;c=1676325447478), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 21:57:29,337 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xf9a3b379be600f37: Processing first storage report for DS-c55526a2-33f8-4f76-86f7-9c07a91b4be8 from datanode 26a0f3f3-fab9-4883-9582-fbed1359d57e 2023-02-13 21:57:29,337 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xf9a3b379be600f37: from storage DS-c55526a2-33f8-4f76-86f7-9c07a91b4be8 node DatanodeRegistration(127.0.0.1:40923, datanodeUuid=26a0f3f3-fab9-4883-9582-fbed1359d57e, infoPort=33295, infoSecurePort=0, ipcPort=44075, storageInfo=lv=-57;cid=testClusterID;nsid=117908355;c=1676325447478), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 21:57:29,338 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x9b20fb7efe69b521: Processing first storage report for DS-884bed52-4ff1-436c-a6b8-11d1227b0b63 from datanode 0a762196-68ef-4608-a8ad-5faf41155fc5 2023-02-13 21:57:29,338 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x9b20fb7efe69b521: from storage DS-884bed52-4ff1-436c-a6b8-11d1227b0b63 node DatanodeRegistration(127.0.0.1:34727, datanodeUuid=0a762196-68ef-4608-a8ad-5faf41155fc5, infoPort=35465, infoSecurePort=0, ipcPort=41023, storageInfo=lv=-57;cid=testClusterID;nsid=117908355;c=1676325447478), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 21:57:29,484 DEBUG [Listener at localhost.localdomain/44075] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822 2023-02-13 21:57:29,533 INFO [Listener at localhost.localdomain/44075] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/cluster_c8eab63c-2879-73ac-9f0f-8254d034d5a9/zookeeper_0, clientPort=58580, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/cluster_c8eab63c-2879-73ac-9f0f-8254d034d5a9/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/cluster_c8eab63c-2879-73ac-9f0f-8254d034d5a9/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-13 21:57:29,543 INFO [Listener at localhost.localdomain/44075] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=58580 2023-02-13 21:57:29,550 INFO [Listener at localhost.localdomain/44075] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 21:57:29,552 INFO [Listener at localhost.localdomain/44075] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 21:57:30,187 INFO [Listener at localhost.localdomain/44075] util.FSUtils(479): Created version file at hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750 with version=8 2023-02-13 21:57:30,187 INFO [Listener at localhost.localdomain/44075] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/hbase-staging 2023-02-13 21:57:30,517 INFO [Listener at localhost.localdomain/44075] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-13 21:57:30,940 INFO [Listener at localhost.localdomain/44075] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=18 2023-02-13 21:57:30,964 INFO [Listener at localhost.localdomain/44075] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 21:57:30,964 INFO [Listener at localhost.localdomain/44075] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 21:57:30,965 INFO [Listener at localhost.localdomain/44075] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 21:57:30,965 INFO [Listener at localhost.localdomain/44075] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 21:57:30,965 INFO [Listener at localhost.localdomain/44075] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 21:57:31,098 INFO [Listener at localhost.localdomain/44075] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 21:57:31,169 DEBUG [Listener at localhost.localdomain/44075] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-02-13 21:57:31,245 INFO [Listener at localhost.localdomain/44075] ipc.NettyRpcServer(120): Bind to /148.251.75.209:46557 2023-02-13 21:57:31,255 INFO [Listener at localhost.localdomain/44075] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 21:57:31,257 INFO [Listener at localhost.localdomain/44075] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 21:57:31,272 INFO [Listener at localhost.localdomain/44075] zookeeper.RecoverableZooKeeper(93): Process identifier=master:46557 connecting to ZooKeeper ensemble=127.0.0.1:58580 2023-02-13 21:57:31,304 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): master:465570x0, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 21:57:31,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:46557-0x1015dd830e00000 connected 2023-02-13 21:57:31,334 DEBUG [Listener at localhost.localdomain/44075] zookeeper.ZKUtil(164): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 21:57:31,335 DEBUG [Listener at localhost.localdomain/44075] zookeeper.ZKUtil(164): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 21:57:31,338 DEBUG [Listener at localhost.localdomain/44075] zookeeper.ZKUtil(164): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 21:57:31,347 DEBUG [Listener at localhost.localdomain/44075] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46557 2023-02-13 21:57:31,347 DEBUG [Listener at localhost.localdomain/44075] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46557 2023-02-13 21:57:31,348 DEBUG [Listener at localhost.localdomain/44075] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46557 2023-02-13 21:57:31,348 DEBUG [Listener at localhost.localdomain/44075] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46557 2023-02-13 21:57:31,348 DEBUG [Listener at localhost.localdomain/44075] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46557 2023-02-13 21:57:31,353 INFO [Listener at localhost.localdomain/44075] master.HMaster(439): hbase.rootdir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750, hbase.cluster.distributed=false 2023-02-13 21:57:31,408 INFO [Listener at localhost.localdomain/44075] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=18 2023-02-13 21:57:31,408 INFO [Listener at localhost.localdomain/44075] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 21:57:31,408 INFO [Listener at localhost.localdomain/44075] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 21:57:31,409 INFO [Listener at localhost.localdomain/44075] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 21:57:31,409 INFO [Listener at localhost.localdomain/44075] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 21:57:31,409 INFO [Listener at localhost.localdomain/44075] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 21:57:31,413 INFO [Listener at localhost.localdomain/44075] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 21:57:31,416 INFO [Listener at localhost.localdomain/44075] ipc.NettyRpcServer(120): Bind to /148.251.75.209:36339 2023-02-13 21:57:31,418 INFO [Listener at localhost.localdomain/44075] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 21:57:31,422 INFO [Listener at localhost.localdomain/44075] mob.MobFileCache(124): MobFileCache disabled 2023-02-13 21:57:31,423 INFO [Listener at localhost.localdomain/44075] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 21:57:31,425 INFO [Listener at localhost.localdomain/44075] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 21:57:31,426 INFO [Listener at localhost.localdomain/44075] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:36339 connecting to ZooKeeper ensemble=127.0.0.1:58580 2023-02-13 21:57:31,431 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): regionserver:363390x0, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 21:57:31,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:36339-0x1015dd830e00001 connected 2023-02-13 21:57:31,432 DEBUG [Listener at localhost.localdomain/44075] zookeeper.ZKUtil(164): regionserver:36339-0x1015dd830e00001, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 21:57:31,434 DEBUG [Listener at localhost.localdomain/44075] zookeeper.ZKUtil(164): regionserver:36339-0x1015dd830e00001, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 21:57:31,435 DEBUG [Listener at localhost.localdomain/44075] zookeeper.ZKUtil(164): regionserver:36339-0x1015dd830e00001, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 21:57:31,435 DEBUG [Listener at localhost.localdomain/44075] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36339 2023-02-13 21:57:31,436 DEBUG [Listener at localhost.localdomain/44075] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36339 2023-02-13 21:57:31,436 DEBUG [Listener at localhost.localdomain/44075] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36339 2023-02-13 21:57:31,437 DEBUG [Listener at localhost.localdomain/44075] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36339 2023-02-13 21:57:31,438 DEBUG [Listener at localhost.localdomain/44075] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36339 2023-02-13 21:57:31,449 INFO [Listener at localhost.localdomain/44075] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=18 2023-02-13 21:57:31,449 INFO [Listener at localhost.localdomain/44075] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 21:57:31,449 INFO [Listener at localhost.localdomain/44075] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 21:57:31,450 INFO [Listener at localhost.localdomain/44075] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 21:57:31,450 INFO [Listener at localhost.localdomain/44075] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 21:57:31,450 INFO [Listener at localhost.localdomain/44075] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 21:57:31,450 INFO [Listener at localhost.localdomain/44075] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 21:57:31,451 INFO [Listener at localhost.localdomain/44075] ipc.NettyRpcServer(120): Bind to /148.251.75.209:45015 2023-02-13 21:57:31,452 INFO [Listener at localhost.localdomain/44075] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 21:57:31,453 INFO [Listener at localhost.localdomain/44075] mob.MobFileCache(124): MobFileCache disabled 2023-02-13 21:57:31,454 INFO [Listener at localhost.localdomain/44075] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 21:57:31,456 INFO [Listener at localhost.localdomain/44075] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 21:57:31,458 INFO [Listener at localhost.localdomain/44075] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:45015 connecting to ZooKeeper ensemble=127.0.0.1:58580 2023-02-13 21:57:31,461 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): regionserver:450150x0, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 21:57:31,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:45015-0x1015dd830e00002 connected 2023-02-13 21:57:31,463 DEBUG [Listener at localhost.localdomain/44075] zookeeper.ZKUtil(164): regionserver:45015-0x1015dd830e00002, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 21:57:31,464 DEBUG [Listener at localhost.localdomain/44075] zookeeper.ZKUtil(164): regionserver:45015-0x1015dd830e00002, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 21:57:31,464 DEBUG [Listener at localhost.localdomain/44075] zookeeper.ZKUtil(164): regionserver:45015-0x1015dd830e00002, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 21:57:31,465 DEBUG [Listener at localhost.localdomain/44075] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=45015 2023-02-13 21:57:31,465 DEBUG [Listener at localhost.localdomain/44075] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=45015 2023-02-13 21:57:31,466 DEBUG [Listener at localhost.localdomain/44075] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=45015 2023-02-13 21:57:31,467 DEBUG [Listener at localhost.localdomain/44075] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=45015 2023-02-13 21:57:31,467 DEBUG [Listener at localhost.localdomain/44075] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=45015 2023-02-13 21:57:31,481 INFO [Listener at localhost.localdomain/44075] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=18 2023-02-13 21:57:31,481 INFO [Listener at localhost.localdomain/44075] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 21:57:31,482 INFO [Listener at localhost.localdomain/44075] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 21:57:31,482 INFO [Listener at localhost.localdomain/44075] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 21:57:31,482 INFO [Listener at localhost.localdomain/44075] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 21:57:31,482 INFO [Listener at localhost.localdomain/44075] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 21:57:31,482 INFO [Listener at localhost.localdomain/44075] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 21:57:31,484 INFO [Listener at localhost.localdomain/44075] ipc.NettyRpcServer(120): Bind to /148.251.75.209:39523 2023-02-13 21:57:31,484 INFO [Listener at localhost.localdomain/44075] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 21:57:31,485 INFO [Listener at localhost.localdomain/44075] mob.MobFileCache(124): MobFileCache disabled 2023-02-13 21:57:31,486 INFO [Listener at localhost.localdomain/44075] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 21:57:31,487 INFO [Listener at localhost.localdomain/44075] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 21:57:31,489 INFO [Listener at localhost.localdomain/44075] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:39523 connecting to ZooKeeper ensemble=127.0.0.1:58580 2023-02-13 21:57:31,493 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): regionserver:395230x0, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 21:57:31,494 DEBUG [Listener at localhost.localdomain/44075] zookeeper.ZKUtil(164): regionserver:395230x0, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 21:57:31,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:39523-0x1015dd830e00003 connected 2023-02-13 21:57:31,494 DEBUG [Listener at localhost.localdomain/44075] zookeeper.ZKUtil(164): regionserver:39523-0x1015dd830e00003, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 21:57:31,495 DEBUG [Listener at localhost.localdomain/44075] zookeeper.ZKUtil(164): regionserver:39523-0x1015dd830e00003, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 21:57:31,496 DEBUG [Listener at localhost.localdomain/44075] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39523 2023-02-13 21:57:31,496 DEBUG [Listener at localhost.localdomain/44075] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39523 2023-02-13 21:57:31,497 DEBUG [Listener at localhost.localdomain/44075] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39523 2023-02-13 21:57:31,497 DEBUG [Listener at localhost.localdomain/44075] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39523 2023-02-13 21:57:31,497 DEBUG [Listener at localhost.localdomain/44075] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39523 2023-02-13 21:57:31,499 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,46557,1676325450344 2023-02-13 21:57:31,509 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 21:57:31,510 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,46557,1676325450344 2023-02-13 21:57:31,529 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): regionserver:45015-0x1015dd830e00002, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 21:57:31,529 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): regionserver:36339-0x1015dd830e00001, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 21:57:31,529 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): regionserver:39523-0x1015dd830e00003, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 21:57:31,529 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 21:57:31,530 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 21:57:31,530 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 21:57:31,531 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,46557,1676325450344 from backup master directory 2023-02-13 21:57:31,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 21:57:31,534 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,46557,1676325450344 2023-02-13 21:57:31,535 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 21:57:31,536 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 21:57:31,536 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase20.apache.org,46557,1676325450344 2023-02-13 21:57:31,538 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-02-13 21:57:31,539 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-02-13 21:57:31,629 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/hbase.id with ID: b91fcacb-c7d0-46e6-8547-6f4f4fdb25e7 2023-02-13 21:57:31,687 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 21:57:31,701 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 21:57:31,760 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x7bbaed0e to 127.0.0.1:58580 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 21:57:31,790 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@58832c1a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 21:57:31,809 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 21:57:31,810 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-13 21:57:31,824 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-02-13 21:57:31,824 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-02-13 21:57:31,826 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-13 21:57:31,829 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-13 21:57:31,830 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 21:57:31,858 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/MasterData/data/master/store-tmp 2023-02-13 21:57:31,926 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:31,926 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 21:57:31,927 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 21:57:31,927 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 21:57:31,927 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 21:57:31,927 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 21:57:31,927 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 21:57:31,927 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676325451926Disabling compacts and flushes for region at 1676325451926Disabling writes for close at 1676325451927 (+1 ms)Writing region close event to WAL at 1676325451927Closed at 1676325451927 2023-02-13 21:57:31,928 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/MasterData/WALs/jenkins-hbase20.apache.org,46557,1676325450344 2023-02-13 21:57:31,948 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C46557%2C1676325450344, suffix=, logDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/MasterData/WALs/jenkins-hbase20.apache.org,46557,1676325450344, archiveDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/MasterData/oldWALs, maxLogs=10 2023-02-13 21:57:32,007 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34727,DS-0adfe1ee-30f2-40dc-88b6-d68d1d5073c7,DISK] 2023-02-13 21:57:32,007 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36475,DS-1ba6ad40-f2ea-4538-8465-3514bd0a9251,DISK] 2023-02-13 21:57:32,007 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40923,DS-951ea18f-1dbb-4a4d-8960-2e8bc8a798dd,DISK] 2023-02-13 21:57:32,017 DEBUG [RS-EventLoopGroup-5-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 21:57:32,080 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/MasterData/WALs/jenkins-hbase20.apache.org,46557,1676325450344/jenkins-hbase20.apache.org%2C46557%2C1676325450344.1676325451957 2023-02-13 21:57:32,081 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36475,DS-1ba6ad40-f2ea-4538-8465-3514bd0a9251,DISK], DatanodeInfoWithStorage[127.0.0.1:34727,DS-0adfe1ee-30f2-40dc-88b6-d68d1d5073c7,DISK], DatanodeInfoWithStorage[127.0.0.1:40923,DS-951ea18f-1dbb-4a4d-8960-2e8bc8a798dd,DISK]] 2023-02-13 21:57:32,081 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-13 21:57:32,082 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:32,084 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 21:57:32,086 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 21:57:32,136 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-13 21:57:32,142 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-13 21:57:32,162 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-13 21:57:32,172 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:32,177 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-13 21:57:32,179 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-13 21:57:32,194 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 21:57:32,198 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 21:57:32,199 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71439295, jitterRate=0.0645284503698349}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:32,200 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676325452108Initializing all the Stores at 1676325452110 (+2 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676325452110Cleaning up temporary data from old regions at 1676325452186 (+76 ms)Cleaning up detritus from prior splits at 1676325452186Region opened successfully at 1676325452200 (+14 ms) 2023-02-13 21:57:32,201 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-13 21:57:32,219 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-13 21:57:32,219 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-13 21:57:32,221 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-13 21:57:32,222 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-02-13 21:57:32,256 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 33 msec 2023-02-13 21:57:32,256 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-13 21:57:32,285 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-13 21:57:32,290 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-13 21:57:32,317 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-13 21:57:32,321 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-13 21:57:32,322 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-13 21:57:32,328 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-13 21:57:32,332 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-13 21:57:32,334 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 21:57:32,336 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-13 21:57:32,337 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-13 21:57:32,353 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-13 21:57:32,358 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): regionserver:36339-0x1015dd830e00001, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 21:57:32,358 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 21:57:32,358 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): regionserver:45015-0x1015dd830e00002, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 21:57:32,358 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 21:57:32,358 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): regionserver:39523-0x1015dd830e00003, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 21:57:32,359 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase20.apache.org,46557,1676325450344, sessionid=0x1015dd830e00000, setting cluster-up flag (Was=false) 2023-02-13 21:57:32,373 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 21:57:32,377 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-13 21:57:32,379 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,46557,1676325450344 2023-02-13 21:57:32,383 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 21:57:32,387 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-13 21:57:32,388 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,46557,1676325450344 2023-02-13 21:57:32,390 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp 2023-02-13 21:57:32,401 INFO [RS:2;jenkins-hbase20:39523] regionserver.HRegionServer(952): ClusterId : b91fcacb-c7d0-46e6-8547-6f4f4fdb25e7 2023-02-13 21:57:32,401 INFO [RS:1;jenkins-hbase20:45015] regionserver.HRegionServer(952): ClusterId : b91fcacb-c7d0-46e6-8547-6f4f4fdb25e7 2023-02-13 21:57:32,401 INFO [RS:0;jenkins-hbase20:36339] regionserver.HRegionServer(952): ClusterId : b91fcacb-c7d0-46e6-8547-6f4f4fdb25e7 2023-02-13 21:57:32,406 DEBUG [RS:0;jenkins-hbase20:36339] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 21:57:32,406 DEBUG [RS:1;jenkins-hbase20:45015] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 21:57:32,406 DEBUG [RS:2;jenkins-hbase20:39523] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 21:57:32,412 DEBUG [RS:1;jenkins-hbase20:45015] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 21:57:32,412 DEBUG [RS:0;jenkins-hbase20:36339] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 21:57:32,412 DEBUG [RS:2;jenkins-hbase20:39523] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 21:57:32,412 DEBUG [RS:0;jenkins-hbase20:36339] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 21:57:32,412 DEBUG [RS:1;jenkins-hbase20:45015] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 21:57:32,412 DEBUG [RS:2;jenkins-hbase20:39523] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 21:57:32,415 DEBUG [RS:2;jenkins-hbase20:39523] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 21:57:32,415 DEBUG [RS:0;jenkins-hbase20:36339] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 21:57:32,416 DEBUG [RS:1;jenkins-hbase20:45015] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 21:57:32,417 DEBUG [RS:0;jenkins-hbase20:36339] zookeeper.ReadOnlyZKClient(139): Connect 0x4413fc87 to 127.0.0.1:58580 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 21:57:32,419 DEBUG [RS:2;jenkins-hbase20:39523] zookeeper.ReadOnlyZKClient(139): Connect 0x2825b05c to 127.0.0.1:58580 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 21:57:32,419 DEBUG [RS:1;jenkins-hbase20:45015] zookeeper.ReadOnlyZKClient(139): Connect 0x422bdf92 to 127.0.0.1:58580 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 21:57:32,432 DEBUG [RS:2;jenkins-hbase20:39523] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@26e25b28, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 21:57:32,433 DEBUG [RS:2;jenkins-hbase20:39523] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@22dddb2c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-02-13 21:57:32,433 DEBUG [RS:1;jenkins-hbase20:45015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4cec29a7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 21:57:32,433 DEBUG [RS:1;jenkins-hbase20:45015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@48d3b4e2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-02-13 21:57:32,435 DEBUG [RS:0;jenkins-hbase20:36339] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4ad41a75, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 21:57:32,435 DEBUG [RS:0;jenkins-hbase20:36339] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4eeb97f7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-02-13 21:57:32,454 DEBUG [RS:1;jenkins-hbase20:45015] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase20:45015 2023-02-13 21:57:32,454 DEBUG [RS:2;jenkins-hbase20:39523] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase20:39523 2023-02-13 21:57:32,458 INFO [RS:2;jenkins-hbase20:39523] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 21:57:32,458 INFO [RS:2;jenkins-hbase20:39523] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 21:57:32,458 INFO [RS:1;jenkins-hbase20:45015] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 21:57:32,458 DEBUG [RS:2;jenkins-hbase20:39523] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 21:57:32,458 INFO [RS:1;jenkins-hbase20:45015] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 21:57:32,459 DEBUG [RS:1;jenkins-hbase20:45015] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 21:57:32,460 DEBUG [RS:0;jenkins-hbase20:36339] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase20:36339 2023-02-13 21:57:32,460 INFO [RS:0;jenkins-hbase20:36339] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 21:57:32,460 INFO [RS:0;jenkins-hbase20:36339] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 21:57:32,460 DEBUG [RS:0;jenkins-hbase20:36339] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 21:57:32,461 INFO [RS:2;jenkins-hbase20:39523] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase20.apache.org,46557,1676325450344 with isa=jenkins-hbase20.apache.org/148.251.75.209:39523, startcode=1676325451480 2023-02-13 21:57:32,461 INFO [RS:0;jenkins-hbase20:36339] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase20.apache.org,46557,1676325450344 with isa=jenkins-hbase20.apache.org/148.251.75.209:36339, startcode=1676325451407 2023-02-13 21:57:32,461 INFO [RS:1;jenkins-hbase20:45015] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase20.apache.org,46557,1676325450344 with isa=jenkins-hbase20.apache.org/148.251.75.209:45015, startcode=1676325451448 2023-02-13 21:57:32,477 DEBUG [RS:2;jenkins-hbase20:39523] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 21:57:32,477 DEBUG [RS:1;jenkins-hbase20:45015] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 21:57:32,477 DEBUG [RS:0;jenkins-hbase20:36339] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 21:57:32,523 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-13 21:57:32,531 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41121, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 21:57:32,531 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55611, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 21:57:32,531 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43215, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 21:57:32,532 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-02-13 21:57:32,532 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-02-13 21:57:32,533 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-02-13 21:57:32,533 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-02-13 21:57:32,533 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-02-13 21:57:32,533 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-13 21:57:32,533 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-02-13 21:57:32,533 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-13 21:57:32,534 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676325482534 2023-02-13 21:57:32,537 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-13 21:57:32,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46557] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:32,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46557] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:32,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:32,550 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-13 21:57:32,550 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-13 21:57:32,551 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-13 21:57:32,558 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-13 21:57:32,558 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-13 21:57:32,558 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-13 21:57:32,559 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-13 21:57:32,559 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-13 21:57:32,560 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 21:57:32,562 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-13 21:57:32,563 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-13 21:57:32,564 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-13 21:57:32,565 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-13 21:57:32,566 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-13 21:57:32,568 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1676325452567,5,FailOnTimeoutGroup] 2023-02-13 21:57:32,569 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1676325452569,5,FailOnTimeoutGroup] 2023-02-13 21:57:32,569 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 21:57:32,569 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-13 21:57:32,570 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-13 21:57:32,570 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-13 21:57:32,572 DEBUG [RS:0;jenkins-hbase20:36339] regionserver.HRegionServer(2829): Master is not running yet 2023-02-13 21:57:32,572 DEBUG [RS:1;jenkins-hbase20:45015] regionserver.HRegionServer(2829): Master is not running yet 2023-02-13 21:57:32,573 WARN [RS:0;jenkins-hbase20:36339] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-13 21:57:32,572 DEBUG [RS:2;jenkins-hbase20:39523] regionserver.HRegionServer(2829): Master is not running yet 2023-02-13 21:57:32,573 WARN [RS:1;jenkins-hbase20:45015] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-13 21:57:32,573 WARN [RS:2;jenkins-hbase20:39523] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-13 21:57:32,598 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-13 21:57:32,599 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-13 21:57:32,599 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750 2023-02-13 21:57:32,625 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:32,628 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 21:57:32,631 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/meta/1588230740/info 2023-02-13 21:57:32,632 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 21:57:32,633 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:32,633 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 21:57:32,636 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/meta/1588230740/rep_barrier 2023-02-13 21:57:32,636 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 21:57:32,637 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:32,637 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 21:57:32,640 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/meta/1588230740/table 2023-02-13 21:57:32,640 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 21:57:32,641 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:32,644 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/meta/1588230740 2023-02-13 21:57:32,645 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/meta/1588230740 2023-02-13 21:57:32,650 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 21:57:32,653 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 21:57:32,657 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 21:57:32,659 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64394325, jitterRate=-0.04044978320598602}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 21:57:32,659 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: Writing region info on filesystem at 1676325452625Initializing all the Stores at 1676325452627 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676325452627Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676325452627Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676325452628 (+1 ms)Cleaning up temporary data from old regions at 1676325452647 (+19 ms)Cleaning up detritus from prior splits at 1676325452648 (+1 ms)Region opened successfully at 1676325452659 (+11 ms) 2023-02-13 21:57:32,659 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 21:57:32,659 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 21:57:32,659 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 21:57:32,660 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 21:57:32,660 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 21:57:32,661 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 21:57:32,662 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676325452659Disabling compacts and flushes for region at 1676325452659Disabling writes for close at 1676325452660 (+1 ms)Writing region close event to WAL at 1676325452661 (+1 ms)Closed at 1676325452661 2023-02-13 21:57:32,668 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-13 21:57:32,668 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-13 21:57:32,674 INFO [RS:0;jenkins-hbase20:36339] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase20.apache.org,46557,1676325450344 with isa=jenkins-hbase20.apache.org/148.251.75.209:36339, startcode=1676325451407 2023-02-13 21:57:32,674 INFO [RS:2;jenkins-hbase20:39523] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase20.apache.org,46557,1676325450344 with isa=jenkins-hbase20.apache.org/148.251.75.209:39523, startcode=1676325451480 2023-02-13 21:57:32,674 INFO [RS:1;jenkins-hbase20:45015] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase20.apache.org,46557,1676325450344 with isa=jenkins-hbase20.apache.org/148.251.75.209:45015, startcode=1676325451448 2023-02-13 21:57:32,680 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46557] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:32,680 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46557] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:32,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:32,681 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-13 21:57:32,687 DEBUG [RS:2;jenkins-hbase20:39523] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750 2023-02-13 21:57:32,688 DEBUG [RS:0;jenkins-hbase20:36339] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750 2023-02-13 21:57:32,687 DEBUG [RS:1;jenkins-hbase20:45015] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750 2023-02-13 21:57:32,688 DEBUG [RS:0;jenkins-hbase20:36339] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost.localdomain:37775 2023-02-13 21:57:32,688 DEBUG [RS:2;jenkins-hbase20:39523] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost.localdomain:37775 2023-02-13 21:57:32,688 DEBUG [RS:0;jenkins-hbase20:36339] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 21:57:32,688 DEBUG [RS:1;jenkins-hbase20:45015] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost.localdomain:37775 2023-02-13 21:57:32,688 DEBUG [RS:2;jenkins-hbase20:39523] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 21:57:32,688 DEBUG [RS:1;jenkins-hbase20:45015] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 21:57:32,695 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 21:57:32,696 DEBUG [RS:2;jenkins-hbase20:39523] zookeeper.ZKUtil(162): regionserver:39523-0x1015dd830e00003, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:32,696 DEBUG [RS:0;jenkins-hbase20:36339] zookeeper.ZKUtil(162): regionserver:36339-0x1015dd830e00001, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:32,696 DEBUG [RS:1;jenkins-hbase20:45015] zookeeper.ZKUtil(162): regionserver:45015-0x1015dd830e00002, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:32,697 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-13 21:57:32,697 WARN [RS:1;jenkins-hbase20:45015] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 21:57:32,697 WARN [RS:0;jenkins-hbase20:36339] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 21:57:32,698 INFO [RS:1;jenkins-hbase20:45015] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 21:57:32,698 INFO [RS:0;jenkins-hbase20:36339] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 21:57:32,696 WARN [RS:2;jenkins-hbase20:39523] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 21:57:32,698 DEBUG [RS:1;jenkins-hbase20:45015] regionserver.HRegionServer(1947): logDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/WALs/jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:32,699 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,36339,1676325451407] 2023-02-13 21:57:32,699 INFO [RS:2;jenkins-hbase20:39523] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 21:57:32,699 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,45015,1676325451448] 2023-02-13 21:57:32,699 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,39523,1676325451480] 2023-02-13 21:57:32,699 DEBUG [RS:2;jenkins-hbase20:39523] regionserver.HRegionServer(1947): logDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/WALs/jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:32,698 DEBUG [RS:0;jenkins-hbase20:36339] regionserver.HRegionServer(1947): logDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/WALs/jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:32,704 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-13 21:57:32,712 DEBUG [RS:1;jenkins-hbase20:45015] zookeeper.ZKUtil(162): regionserver:45015-0x1015dd830e00002, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:32,712 DEBUG [RS:2;jenkins-hbase20:39523] zookeeper.ZKUtil(162): regionserver:39523-0x1015dd830e00003, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:32,712 DEBUG [RS:0;jenkins-hbase20:36339] zookeeper.ZKUtil(162): regionserver:36339-0x1015dd830e00001, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:32,713 DEBUG [RS:1;jenkins-hbase20:45015] zookeeper.ZKUtil(162): regionserver:45015-0x1015dd830e00002, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:32,713 DEBUG [RS:2;jenkins-hbase20:39523] zookeeper.ZKUtil(162): regionserver:39523-0x1015dd830e00003, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:32,713 DEBUG [RS:0;jenkins-hbase20:36339] zookeeper.ZKUtil(162): regionserver:36339-0x1015dd830e00001, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:32,713 DEBUG [RS:1;jenkins-hbase20:45015] zookeeper.ZKUtil(162): regionserver:45015-0x1015dd830e00002, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:32,713 DEBUG [RS:2;jenkins-hbase20:39523] zookeeper.ZKUtil(162): regionserver:39523-0x1015dd830e00003, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:32,713 DEBUG [RS:0;jenkins-hbase20:36339] zookeeper.ZKUtil(162): regionserver:36339-0x1015dd830e00001, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:32,724 DEBUG [RS:2;jenkins-hbase20:39523] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 21:57:32,724 DEBUG [RS:1;jenkins-hbase20:45015] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 21:57:32,724 DEBUG [RS:0;jenkins-hbase20:36339] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 21:57:32,732 INFO [RS:2;jenkins-hbase20:39523] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 21:57:32,732 INFO [RS:1;jenkins-hbase20:45015] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 21:57:32,732 INFO [RS:0;jenkins-hbase20:36339] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 21:57:32,753 INFO [RS:2;jenkins-hbase20:39523] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 21:57:32,753 INFO [RS:0;jenkins-hbase20:36339] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 21:57:32,753 INFO [RS:1;jenkins-hbase20:45015] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 21:57:32,757 INFO [RS:2;jenkins-hbase20:39523] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 21:57:32,757 INFO [RS:1;jenkins-hbase20:45015] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 21:57:32,757 INFO [RS:0;jenkins-hbase20:36339] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 21:57:32,758 INFO [RS:1;jenkins-hbase20:45015] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 21:57:32,757 INFO [RS:2;jenkins-hbase20:39523] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 21:57:32,758 INFO [RS:0;jenkins-hbase20:36339] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 21:57:32,759 INFO [RS:2;jenkins-hbase20:39523] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 21:57:32,759 INFO [RS:1;jenkins-hbase20:45015] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 21:57:32,759 INFO [RS:0;jenkins-hbase20:36339] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 21:57:32,767 INFO [RS:2;jenkins-hbase20:39523] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 21:57:32,767 INFO [RS:1;jenkins-hbase20:45015] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 21:57:32,767 INFO [RS:0;jenkins-hbase20:36339] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 21:57:32,767 DEBUG [RS:2;jenkins-hbase20:39523] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-13 21:57:32,767 DEBUG [RS:1;jenkins-hbase20:45015] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-13 21:57:32,767 DEBUG [RS:0;jenkins-hbase20:36339] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-13 21:57:32,767 DEBUG [RS:1;jenkins-hbase20:45015] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-13 21:57:32,767 DEBUG [RS:2;jenkins-hbase20:39523] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-13 21:57:32,768 DEBUG [RS:1;jenkins-hbase20:45015] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-13 21:57:32,768 DEBUG [RS:2;jenkins-hbase20:39523] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-13 21:57:32,768 DEBUG [RS:0;jenkins-hbase20:36339] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-13 21:57:32,768 DEBUG [RS:2;jenkins-hbase20:39523] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-13 21:57:32,768 DEBUG [RS:0;jenkins-hbase20:36339] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-13 21:57:32,768 DEBUG [RS:2;jenkins-hbase20:39523] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-13 21:57:32,768 DEBUG [RS:1;jenkins-hbase20:45015] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-13 21:57:32,768 DEBUG [RS:2;jenkins-hbase20:39523] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-02-13 21:57:32,768 DEBUG [RS:0;jenkins-hbase20:36339] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-13 21:57:32,768 DEBUG [RS:2;jenkins-hbase20:39523] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-13 21:57:32,768 DEBUG [RS:0;jenkins-hbase20:36339] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-13 21:57:32,768 DEBUG [RS:1;jenkins-hbase20:45015] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-13 21:57:32,768 DEBUG [RS:0;jenkins-hbase20:36339] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-02-13 21:57:32,769 DEBUG [RS:1;jenkins-hbase20:45015] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-02-13 21:57:32,769 DEBUG [RS:0;jenkins-hbase20:36339] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-13 21:57:32,769 DEBUG [RS:1;jenkins-hbase20:45015] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-13 21:57:32,769 DEBUG [RS:0;jenkins-hbase20:36339] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-13 21:57:32,769 DEBUG [RS:1;jenkins-hbase20:45015] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-13 21:57:32,769 DEBUG [RS:0;jenkins-hbase20:36339] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-13 21:57:32,769 DEBUG [RS:1;jenkins-hbase20:45015] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-13 21:57:32,768 DEBUG [RS:2;jenkins-hbase20:39523] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-13 21:57:32,769 DEBUG [RS:1;jenkins-hbase20:45015] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-13 21:57:32,769 DEBUG [RS:2;jenkins-hbase20:39523] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-13 21:57:32,769 DEBUG [RS:0;jenkins-hbase20:36339] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-13 21:57:32,769 DEBUG [RS:2;jenkins-hbase20:39523] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-13 21:57:32,771 INFO [RS:1;jenkins-hbase20:45015] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 21:57:32,771 INFO [RS:0;jenkins-hbase20:36339] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 21:57:32,771 INFO [RS:1;jenkins-hbase20:45015] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 21:57:32,771 INFO [RS:0;jenkins-hbase20:36339] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 21:57:32,771 INFO [RS:2;jenkins-hbase20:39523] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 21:57:32,771 INFO [RS:1;jenkins-hbase20:45015] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 21:57:32,771 INFO [RS:2;jenkins-hbase20:39523] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 21:57:32,771 INFO [RS:0;jenkins-hbase20:36339] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 21:57:32,771 INFO [RS:2;jenkins-hbase20:39523] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 21:57:32,785 INFO [RS:0;jenkins-hbase20:36339] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 21:57:32,787 INFO [RS:0;jenkins-hbase20:36339] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,36339,1676325451407-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 21:57:32,787 INFO [RS:2;jenkins-hbase20:39523] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 21:57:32,787 INFO [RS:1;jenkins-hbase20:45015] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 21:57:32,788 INFO [RS:2;jenkins-hbase20:39523] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,39523,1676325451480-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 21:57:32,788 INFO [RS:1;jenkins-hbase20:45015] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,45015,1676325451448-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 21:57:32,802 INFO [RS:1;jenkins-hbase20:45015] regionserver.Replication(203): jenkins-hbase20.apache.org,45015,1676325451448 started 2023-02-13 21:57:32,802 INFO [RS:1;jenkins-hbase20:45015] regionserver.HRegionServer(1638): Serving as jenkins-hbase20.apache.org,45015,1676325451448, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:45015, sessionid=0x1015dd830e00002 2023-02-13 21:57:32,807 INFO [RS:0;jenkins-hbase20:36339] regionserver.Replication(203): jenkins-hbase20.apache.org,36339,1676325451407 started 2023-02-13 21:57:32,807 DEBUG [RS:1;jenkins-hbase20:45015] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 21:57:32,807 INFO [RS:2;jenkins-hbase20:39523] regionserver.Replication(203): jenkins-hbase20.apache.org,39523,1676325451480 started 2023-02-13 21:57:32,807 INFO [RS:0;jenkins-hbase20:36339] regionserver.HRegionServer(1638): Serving as jenkins-hbase20.apache.org,36339,1676325451407, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:36339, sessionid=0x1015dd830e00001 2023-02-13 21:57:32,807 INFO [RS:2;jenkins-hbase20:39523] regionserver.HRegionServer(1638): Serving as jenkins-hbase20.apache.org,39523,1676325451480, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:39523, sessionid=0x1015dd830e00003 2023-02-13 21:57:32,807 DEBUG [RS:0;jenkins-hbase20:36339] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 21:57:32,807 DEBUG [RS:1;jenkins-hbase20:45015] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:32,807 DEBUG [RS:0;jenkins-hbase20:36339] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:32,807 DEBUG [RS:2;jenkins-hbase20:39523] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 21:57:32,808 DEBUG [RS:2;jenkins-hbase20:39523] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:32,808 DEBUG [RS:2;jenkins-hbase20:39523] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,39523,1676325451480' 2023-02-13 21:57:32,808 DEBUG [RS:0;jenkins-hbase20:36339] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,36339,1676325451407' 2023-02-13 21:57:32,808 DEBUG [RS:0;jenkins-hbase20:36339] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 21:57:32,808 DEBUG [RS:1;jenkins-hbase20:45015] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,45015,1676325451448' 2023-02-13 21:57:32,809 DEBUG [RS:1;jenkins-hbase20:45015] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 21:57:32,808 DEBUG [RS:2;jenkins-hbase20:39523] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 21:57:32,809 DEBUG [RS:0;jenkins-hbase20:36339] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 21:57:32,809 DEBUG [RS:2;jenkins-hbase20:39523] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 21:57:32,809 DEBUG [RS:1;jenkins-hbase20:45015] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 21:57:32,810 DEBUG [RS:0;jenkins-hbase20:36339] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 21:57:32,810 DEBUG [RS:2;jenkins-hbase20:39523] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 21:57:32,810 DEBUG [RS:1;jenkins-hbase20:45015] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 21:57:32,810 DEBUG [RS:2;jenkins-hbase20:39523] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 21:57:32,810 DEBUG [RS:0;jenkins-hbase20:36339] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 21:57:32,810 DEBUG [RS:2;jenkins-hbase20:39523] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:32,810 DEBUG [RS:1;jenkins-hbase20:45015] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 21:57:32,811 DEBUG [RS:2;jenkins-hbase20:39523] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,39523,1676325451480' 2023-02-13 21:57:32,811 DEBUG [RS:2;jenkins-hbase20:39523] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 21:57:32,810 DEBUG [RS:0;jenkins-hbase20:36339] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:32,811 DEBUG [RS:1;jenkins-hbase20:45015] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:32,811 DEBUG [RS:1;jenkins-hbase20:45015] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,45015,1676325451448' 2023-02-13 21:57:32,811 DEBUG [RS:1;jenkins-hbase20:45015] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 21:57:32,811 DEBUG [RS:0;jenkins-hbase20:36339] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,36339,1676325451407' 2023-02-13 21:57:32,811 DEBUG [RS:0;jenkins-hbase20:36339] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 21:57:32,811 DEBUG [RS:2;jenkins-hbase20:39523] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 21:57:32,811 DEBUG [RS:1;jenkins-hbase20:45015] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 21:57:32,811 DEBUG [RS:0;jenkins-hbase20:36339] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 21:57:32,812 DEBUG [RS:2;jenkins-hbase20:39523] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 21:57:32,812 INFO [RS:2;jenkins-hbase20:39523] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 21:57:32,812 DEBUG [RS:1;jenkins-hbase20:45015] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 21:57:32,812 INFO [RS:2;jenkins-hbase20:39523] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 21:57:32,812 INFO [RS:1;jenkins-hbase20:45015] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 21:57:32,812 DEBUG [RS:0;jenkins-hbase20:36339] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 21:57:32,812 INFO [RS:1;jenkins-hbase20:45015] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 21:57:32,813 INFO [RS:0;jenkins-hbase20:36339] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 21:57:32,813 INFO [RS:0;jenkins-hbase20:36339] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 21:57:32,856 DEBUG [jenkins-hbase20:46557] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-13 21:57:32,860 DEBUG [jenkins-hbase20:46557] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-02-13 21:57:32,865 DEBUG [jenkins-hbase20:46557] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 21:57:32,865 DEBUG [jenkins-hbase20:46557] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 21:57:32,865 DEBUG [jenkins-hbase20:46557] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 21:57:32,865 DEBUG [jenkins-hbase20:46557] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 21:57:32,868 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,36339,1676325451407, state=OPENING 2023-02-13 21:57:32,873 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-13 21:57:32,874 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 21:57:32,875 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 21:57:32,878 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:57:32,927 INFO [RS:0;jenkins-hbase20:36339] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C36339%2C1676325451407, suffix=, logDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/WALs/jenkins-hbase20.apache.org,36339,1676325451407, archiveDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/oldWALs, maxLogs=32 2023-02-13 21:57:32,927 INFO [RS:1;jenkins-hbase20:45015] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C45015%2C1676325451448, suffix=, logDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/WALs/jenkins-hbase20.apache.org,45015,1676325451448, archiveDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/oldWALs, maxLogs=32 2023-02-13 21:57:32,929 INFO [RS:2;jenkins-hbase20:39523] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C39523%2C1676325451480, suffix=, logDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/WALs/jenkins-hbase20.apache.org,39523,1676325451480, archiveDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/oldWALs, maxLogs=32 2023-02-13 21:57:32,954 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40923,DS-951ea18f-1dbb-4a4d-8960-2e8bc8a798dd,DISK] 2023-02-13 21:57:32,954 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34727,DS-0adfe1ee-30f2-40dc-88b6-d68d1d5073c7,DISK] 2023-02-13 21:57:32,954 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36475,DS-1ba6ad40-f2ea-4538-8465-3514bd0a9251,DISK] 2023-02-13 21:57:32,966 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34727,DS-0adfe1ee-30f2-40dc-88b6-d68d1d5073c7,DISK] 2023-02-13 21:57:32,966 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36475,DS-1ba6ad40-f2ea-4538-8465-3514bd0a9251,DISK] 2023-02-13 21:57:32,972 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34727,DS-0adfe1ee-30f2-40dc-88b6-d68d1d5073c7,DISK] 2023-02-13 21:57:32,975 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36475,DS-1ba6ad40-f2ea-4538-8465-3514bd0a9251,DISK] 2023-02-13 21:57:32,975 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40923,DS-951ea18f-1dbb-4a4d-8960-2e8bc8a798dd,DISK] 2023-02-13 21:57:32,989 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40923,DS-951ea18f-1dbb-4a4d-8960-2e8bc8a798dd,DISK] 2023-02-13 21:57:32,991 INFO [RS:2;jenkins-hbase20:39523] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/WALs/jenkins-hbase20.apache.org,39523,1676325451480/jenkins-hbase20.apache.org%2C39523%2C1676325451480.1676325452932 2023-02-13 21:57:32,991 INFO [RS:1;jenkins-hbase20:45015] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/WALs/jenkins-hbase20.apache.org,45015,1676325451448/jenkins-hbase20.apache.org%2C45015%2C1676325451448.1676325452932 2023-02-13 21:57:32,991 DEBUG [RS:2;jenkins-hbase20:39523] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34727,DS-0adfe1ee-30f2-40dc-88b6-d68d1d5073c7,DISK], DatanodeInfoWithStorage[127.0.0.1:36475,DS-1ba6ad40-f2ea-4538-8465-3514bd0a9251,DISK], DatanodeInfoWithStorage[127.0.0.1:40923,DS-951ea18f-1dbb-4a4d-8960-2e8bc8a798dd,DISK]] 2023-02-13 21:57:32,991 DEBUG [RS:1;jenkins-hbase20:45015] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34727,DS-0adfe1ee-30f2-40dc-88b6-d68d1d5073c7,DISK], DatanodeInfoWithStorage[127.0.0.1:36475,DS-1ba6ad40-f2ea-4538-8465-3514bd0a9251,DISK], DatanodeInfoWithStorage[127.0.0.1:40923,DS-951ea18f-1dbb-4a4d-8960-2e8bc8a798dd,DISK]] 2023-02-13 21:57:32,996 INFO [RS:0;jenkins-hbase20:36339] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/WALs/jenkins-hbase20.apache.org,36339,1676325451407/jenkins-hbase20.apache.org%2C36339%2C1676325451407.1676325452932 2023-02-13 21:57:32,997 DEBUG [RS:0;jenkins-hbase20:36339] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34727,DS-0adfe1ee-30f2-40dc-88b6-d68d1d5073c7,DISK], DatanodeInfoWithStorage[127.0.0.1:36475,DS-1ba6ad40-f2ea-4538-8465-3514bd0a9251,DISK], DatanodeInfoWithStorage[127.0.0.1:40923,DS-951ea18f-1dbb-4a4d-8960-2e8bc8a798dd,DISK]] 2023-02-13 21:57:33,058 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:33,060 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 21:57:33,063 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60452, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 21:57:33,074 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-13 21:57:33,075 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 21:57:33,079 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C36339%2C1676325451407.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/WALs/jenkins-hbase20.apache.org,36339,1676325451407, archiveDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/oldWALs, maxLogs=32 2023-02-13 21:57:33,099 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36475,DS-1ba6ad40-f2ea-4538-8465-3514bd0a9251,DISK] 2023-02-13 21:57:33,099 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34727,DS-0adfe1ee-30f2-40dc-88b6-d68d1d5073c7,DISK] 2023-02-13 21:57:33,100 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40923,DS-951ea18f-1dbb-4a4d-8960-2e8bc8a798dd,DISK] 2023-02-13 21:57:33,105 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/WALs/jenkins-hbase20.apache.org,36339,1676325451407/jenkins-hbase20.apache.org%2C36339%2C1676325451407.meta.1676325453081.meta 2023-02-13 21:57:33,106 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34727,DS-0adfe1ee-30f2-40dc-88b6-d68d1d5073c7,DISK], DatanodeInfoWithStorage[127.0.0.1:36475,DS-1ba6ad40-f2ea-4538-8465-3514bd0a9251,DISK], DatanodeInfoWithStorage[127.0.0.1:40923,DS-951ea18f-1dbb-4a4d-8960-2e8bc8a798dd,DISK]] 2023-02-13 21:57:33,106 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-13 21:57:33,108 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-13 21:57:33,126 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-13 21:57:33,130 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-13 21:57:33,134 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-13 21:57:33,135 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:33,135 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-13 21:57:33,135 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-13 21:57:33,139 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 21:57:33,142 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/meta/1588230740/info 2023-02-13 21:57:33,142 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/meta/1588230740/info 2023-02-13 21:57:33,143 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 21:57:33,145 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:33,146 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 21:57:33,148 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/meta/1588230740/rep_barrier 2023-02-13 21:57:33,148 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/meta/1588230740/rep_barrier 2023-02-13 21:57:33,148 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 21:57:33,150 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:33,150 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 21:57:33,154 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/meta/1588230740/table 2023-02-13 21:57:33,154 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/meta/1588230740/table 2023-02-13 21:57:33,155 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 21:57:33,156 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:33,160 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/meta/1588230740 2023-02-13 21:57:33,163 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/meta/1588230740 2023-02-13 21:57:33,170 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 21:57:33,174 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 21:57:33,176 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72439612, jitterRate=0.079434335231781}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 21:57:33,177 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676325453135Writing region info on filesystem at 1676325453135Initializing all the Stores at 1676325453138 (+3 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676325453139 (+1 ms)Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676325453139Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676325453139Cleaning up temporary data from old regions at 1676325453167 (+28 ms)Cleaning up detritus from prior splits at 1676325453167Running coprocessor post-open hooks at 1676325453177 (+10 ms)Region opened successfully at 1676325453177 2023-02-13 21:57:33,191 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1676325453052 2023-02-13 21:57:33,213 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-13 21:57:33,215 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-13 21:57:33,215 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,36339,1676325451407, state=OPEN 2023-02-13 21:57:33,219 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-13 21:57:33,219 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 21:57:33,227 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-13 21:57:33,227 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,36339,1676325451407 in 341 msec 2023-02-13 21:57:33,234 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-13 21:57:33,234 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 548 msec 2023-02-13 21:57:33,243 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 790 msec 2023-02-13 21:57:33,244 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676325451525, completionTime=-1 2023-02-13 21:57:33,244 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-13 21:57:33,244 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-13 21:57:33,311 DEBUG [hconnection-0x43e34ef3-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 21:57:33,315 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60468, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 21:57:33,345 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-13 21:57:33,345 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676325513345 2023-02-13 21:57:33,345 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676325573345 2023-02-13 21:57:33,345 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 101 msec 2023-02-13 21:57:33,375 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,46557,1676325450344-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 21:57:33,376 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,46557,1676325450344-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 21:57:33,376 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,46557,1676325450344-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 21:57:33,377 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:46557, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 21:57:33,378 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-13 21:57:33,385 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-13 21:57:33,391 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-13 21:57:33,393 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2138): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-13 21:57:33,402 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-13 21:57:33,406 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 21:57:33,408 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 21:57:33,430 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/hbase/namespace/5a6c89db834bbded2000d426b959f122 2023-02-13 21:57:33,432 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/hbase/namespace/5a6c89db834bbded2000d426b959f122 empty. 2023-02-13 21:57:33,433 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/hbase/namespace/5a6c89db834bbded2000d426b959f122 2023-02-13 21:57:33,433 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-13 21:57:33,472 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-13 21:57:33,474 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => 5a6c89db834bbded2000d426b959f122, NAME => 'hbase:namespace,,1676325453392.5a6c89db834bbded2000d426b959f122.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp 2023-02-13 21:57:33,494 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676325453392.5a6c89db834bbded2000d426b959f122.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:33,495 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing 5a6c89db834bbded2000d426b959f122, disabling compactions & flushes 2023-02-13 21:57:33,495 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676325453392.5a6c89db834bbded2000d426b959f122. 2023-02-13 21:57:33,495 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676325453392.5a6c89db834bbded2000d426b959f122. 2023-02-13 21:57:33,495 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676325453392.5a6c89db834bbded2000d426b959f122. after waiting 0 ms 2023-02-13 21:57:33,495 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676325453392.5a6c89db834bbded2000d426b959f122. 2023-02-13 21:57:33,495 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676325453392.5a6c89db834bbded2000d426b959f122. 2023-02-13 21:57:33,495 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for 5a6c89db834bbded2000d426b959f122: Waiting for close lock at 1676325453495Disabling compacts and flushes for region at 1676325453495Disabling writes for close at 1676325453495Writing region close event to WAL at 1676325453495Closed at 1676325453495 2023-02-13 21:57:33,501 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 21:57:33,515 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1676325453392.5a6c89db834bbded2000d426b959f122.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676325453504"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325453504"}]},"ts":"1676325453504"} 2023-02-13 21:57:33,541 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 21:57:33,543 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 21:57:33,548 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325453543"}]},"ts":"1676325453543"} 2023-02-13 21:57:33,553 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-13 21:57:33,557 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-02-13 21:57:33,558 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 21:57:33,558 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 21:57:33,558 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 21:57:33,558 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 21:57:33,560 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=5a6c89db834bbded2000d426b959f122, ASSIGN}] 2023-02-13 21:57:33,563 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=5a6c89db834bbded2000d426b959f122, ASSIGN 2023-02-13 21:57:33,565 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=5a6c89db834bbded2000d426b959f122, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,39523,1676325451480; forceNewPlan=false, retain=false 2023-02-13 21:57:33,717 INFO [jenkins-hbase20:46557] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 21:57:33,718 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=5a6c89db834bbded2000d426b959f122, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:33,718 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676325453392.5a6c89db834bbded2000d426b959f122.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676325453718"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325453718"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325453718"}]},"ts":"1676325453718"} 2023-02-13 21:57:33,726 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 5a6c89db834bbded2000d426b959f122, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:57:33,880 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:33,881 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 21:57:33,887 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55812, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 21:57:33,896 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676325453392.5a6c89db834bbded2000d426b959f122. 2023-02-13 21:57:33,896 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 5a6c89db834bbded2000d426b959f122, NAME => 'hbase:namespace,,1676325453392.5a6c89db834bbded2000d426b959f122.', STARTKEY => '', ENDKEY => ''} 2023-02-13 21:57:33,897 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 5a6c89db834bbded2000d426b959f122 2023-02-13 21:57:33,897 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676325453392.5a6c89db834bbded2000d426b959f122.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:33,897 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 5a6c89db834bbded2000d426b959f122 2023-02-13 21:57:33,897 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 5a6c89db834bbded2000d426b959f122 2023-02-13 21:57:33,899 INFO [StoreOpener-5a6c89db834bbded2000d426b959f122-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 5a6c89db834bbded2000d426b959f122 2023-02-13 21:57:33,901 DEBUG [StoreOpener-5a6c89db834bbded2000d426b959f122-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/namespace/5a6c89db834bbded2000d426b959f122/info 2023-02-13 21:57:33,901 DEBUG [StoreOpener-5a6c89db834bbded2000d426b959f122-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/namespace/5a6c89db834bbded2000d426b959f122/info 2023-02-13 21:57:33,902 INFO [StoreOpener-5a6c89db834bbded2000d426b959f122-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5a6c89db834bbded2000d426b959f122 columnFamilyName info 2023-02-13 21:57:33,902 INFO [StoreOpener-5a6c89db834bbded2000d426b959f122-1] regionserver.HStore(310): Store=5a6c89db834bbded2000d426b959f122/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:33,904 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/namespace/5a6c89db834bbded2000d426b959f122 2023-02-13 21:57:33,906 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/namespace/5a6c89db834bbded2000d426b959f122 2023-02-13 21:57:33,910 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 5a6c89db834bbded2000d426b959f122 2023-02-13 21:57:33,914 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/namespace/5a6c89db834bbded2000d426b959f122/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 21:57:33,915 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 5a6c89db834bbded2000d426b959f122; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74302537, jitterRate=0.10719408094882965}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:33,915 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 5a6c89db834bbded2000d426b959f122: Running coprocessor pre-open hook at 1676325453897Writing region info on filesystem at 1676325453897Initializing all the Stores at 1676325453899 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676325453899Cleaning up temporary data from old regions at 1676325453906 (+7 ms)Cleaning up detritus from prior splits at 1676325453907 (+1 ms)Running coprocessor post-open hooks at 1676325453915 (+8 ms)Region opened successfully at 1676325453915 2023-02-13 21:57:33,920 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676325453392.5a6c89db834bbded2000d426b959f122., pid=6, masterSystemTime=1676325453880 2023-02-13 21:57:33,925 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676325453392.5a6c89db834bbded2000d426b959f122. 2023-02-13 21:57:33,925 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676325453392.5a6c89db834bbded2000d426b959f122. 2023-02-13 21:57:33,927 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=5a6c89db834bbded2000d426b959f122, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:33,928 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676325453392.5a6c89db834bbded2000d426b959f122.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676325453926"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325453926"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325453926"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325453926"}]},"ts":"1676325453926"} 2023-02-13 21:57:33,938 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-13 21:57:33,938 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 5a6c89db834bbded2000d426b959f122, server=jenkins-hbase20.apache.org,39523,1676325451480 in 206 msec 2023-02-13 21:57:33,942 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-13 21:57:33,943 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=5a6c89db834bbded2000d426b959f122, ASSIGN in 378 msec 2023-02-13 21:57:33,944 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 21:57:33,945 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325453944"}]},"ts":"1676325453944"} 2023-02-13 21:57:33,947 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-13 21:57:33,950 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 21:57:33,953 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 556 msec 2023-02-13 21:57:34,007 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-13 21:57:34,008 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-13 21:57:34,008 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 21:57:34,026 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 21:57:34,027 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55818, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 21:57:34,040 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-13 21:57:34,058 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-13 21:57:34,065 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 31 msec 2023-02-13 21:57:34,074 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-13 21:57:34,086 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-13 21:57:34,090 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 17 msec 2023-02-13 21:57:34,101 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-13 21:57:34,102 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-13 21:57:34,102 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 2.566sec 2023-02-13 21:57:34,106 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-13 21:57:34,107 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-13 21:57:34,108 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-13 21:57:34,109 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,46557,1676325450344-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-13 21:57:34,111 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,46557,1676325450344-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-13 21:57:34,119 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-13 21:57:34,206 DEBUG [Listener at localhost.localdomain/44075] zookeeper.ReadOnlyZKClient(139): Connect 0x0ad20338 to 127.0.0.1:58580 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 21:57:34,211 DEBUG [Listener at localhost.localdomain/44075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63c06727, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 21:57:34,225 DEBUG [hconnection-0x1d9ca414-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 21:57:34,238 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34534, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 21:57:34,246 INFO [Listener at localhost.localdomain/44075] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase20.apache.org,46557,1676325450344 2023-02-13 21:57:34,274 INFO [Listener at localhost.localdomain/44075] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=363, OpenFileDescriptor=695, MaxFileDescriptor=60000, SystemLoadAverage=142, ProcessCount=165, AvailableMemoryMB=7283 2023-02-13 21:57:34,287 DEBUG [Listener at localhost.localdomain/44075] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-13 21:57:34,290 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57230, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-13 21:57:34,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.HMaster$4(2102): Client=jenkins//148.251.75.209 create 'testRestoreSnapshot-1676325454277', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-02-13 21:57:34,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:34,310 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676325454277 execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 21:57:34,313 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676325454277 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 21:57:34,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "testRestoreSnapshot-1676325454277" procId is: 9 2023-02-13 21:57:34,321 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:34,321 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:34,321 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:34,321 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:34,322 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:34,322 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375 2023-02-13 21:57:34,322 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70 empty. 2023-02-13 21:57:34,322 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb empty. 2023-02-13 21:57:34,322 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62 empty. 2023-02-13 21:57:34,322 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375 empty. 2023-02-13 21:57:34,323 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d empty. 2023-02-13 21:57:34,325 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:34,325 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d empty. 2023-02-13 21:57:34,325 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375 2023-02-13 21:57:34,325 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:34,325 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:34,326 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:34,326 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:34,326 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676325454277 regions 2023-02-13 21:57:34,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 21:57:34,349 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/.tabledesc/.tableinfo.0000000001 2023-02-13 21:57:34,351 INFO [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-0] regionserver.HRegion(7671): creating {ENCODED => 1f6dc79bf11da8444a0870d779d2fb4d, NAME => 'testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1676325454277', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp 2023-02-13 21:57:34,355 INFO [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-1] regionserver.HRegion(7671): creating {ENCODED => 992f98999ce7b2c13e6adb6eca28ff70, NAME => 'testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1676325454277', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp 2023-02-13 21:57:34,356 INFO [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-2] regionserver.HRegion(7671): creating {ENCODED => 44e821dce55cb29b05989bca574cff62, NAME => 'testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1676325454277', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp 2023-02-13 21:57:34,396 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:34,397 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-0] regionserver.HRegion(1603): Closing 1f6dc79bf11da8444a0870d779d2fb4d, disabling compactions & flushes 2023-02-13 21:57:34,397 INFO [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:34,397 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:34,397 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. after waiting 0 ms 2023-02-13 21:57:34,398 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:34,398 INFO [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:34,398 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-0] regionserver.HRegion(1557): Region close journal for 1f6dc79bf11da8444a0870d779d2fb4d: Waiting for close lock at 1676325454397Disabling compacts and flushes for region at 1676325454397Disabling writes for close at 1676325454398 (+1 ms)Writing region close event to WAL at 1676325454398Closed at 1676325454398 2023-02-13 21:57:34,400 INFO [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-0] regionserver.HRegion(7671): creating {ENCODED => 676fb98d6cc62f0fdbda4ce59aad85cb, NAME => 'testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1676325454277', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp 2023-02-13 21:57:34,405 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:34,406 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-2] regionserver.HRegion(1603): Closing 44e821dce55cb29b05989bca574cff62, disabling compactions & flushes 2023-02-13 21:57:34,406 INFO [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:34,406 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:34,407 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. after waiting 0 ms 2023-02-13 21:57:34,408 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:34,408 INFO [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:34,408 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:34,410 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-2] regionserver.HRegion(1557): Region close journal for 44e821dce55cb29b05989bca574cff62: Waiting for close lock at 1676325454406Disabling compacts and flushes for region at 1676325454406Disabling writes for close at 1676325454408 (+2 ms)Writing region close event to WAL at 1676325454408Closed at 1676325454408 2023-02-13 21:57:34,410 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-1] regionserver.HRegion(1603): Closing 992f98999ce7b2c13e6adb6eca28ff70, disabling compactions & flushes 2023-02-13 21:57:34,411 INFO [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:34,412 INFO [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-2] regionserver.HRegion(7671): creating {ENCODED => 4e757d660af6a5aaa6e2c43f8b91311d, NAME => 'testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1676325454277', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp 2023-02-13 21:57:34,412 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:34,412 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. after waiting 0 ms 2023-02-13 21:57:34,412 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:34,412 INFO [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:34,412 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-1] regionserver.HRegion(1557): Region close journal for 992f98999ce7b2c13e6adb6eca28ff70: Waiting for close lock at 1676325454410Disabling compacts and flushes for region at 1676325454410Disabling writes for close at 1676325454412 (+2 ms)Writing region close event to WAL at 1676325454412Closed at 1676325454412 2023-02-13 21:57:34,413 INFO [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-1] regionserver.HRegion(7671): creating {ENCODED => 251bc68b009792a14e42934b238a6375, NAME => 'testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1676325454277', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp 2023-02-13 21:57:34,437 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:34,438 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-0] regionserver.HRegion(1603): Closing 676fb98d6cc62f0fdbda4ce59aad85cb, disabling compactions & flushes 2023-02-13 21:57:34,438 INFO [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:34,438 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:34,438 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. after waiting 0 ms 2023-02-13 21:57:34,439 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:34,439 INFO [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:34,439 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-0] regionserver.HRegion(1557): Region close journal for 676fb98d6cc62f0fdbda4ce59aad85cb: Waiting for close lock at 1676325454438Disabling compacts and flushes for region at 1676325454438Disabling writes for close at 1676325454439 (+1 ms)Writing region close event to WAL at 1676325454439Closed at 1676325454439 2023-02-13 21:57:34,445 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:34,445 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:34,446 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-2] regionserver.HRegion(1603): Closing 4e757d660af6a5aaa6e2c43f8b91311d, disabling compactions & flushes 2023-02-13 21:57:34,446 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-1] regionserver.HRegion(1603): Closing 251bc68b009792a14e42934b238a6375, disabling compactions & flushes 2023-02-13 21:57:34,446 INFO [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:34,446 INFO [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:34,446 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:34,446 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:34,446 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. after waiting 0 ms 2023-02-13 21:57:34,446 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. after waiting 0 ms 2023-02-13 21:57:34,446 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:34,447 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:34,447 INFO [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:34,447 INFO [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:34,447 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-2] regionserver.HRegion(1557): Region close journal for 4e757d660af6a5aaa6e2c43f8b91311d: Waiting for close lock at 1676325454446Disabling compacts and flushes for region at 1676325454446Disabling writes for close at 1676325454446Writing region close event to WAL at 1676325454447 (+1 ms)Closed at 1676325454447 2023-02-13 21:57:34,447 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676325454277-pool-1] regionserver.HRegion(1557): Region close journal for 251bc68b009792a14e42934b238a6375: Waiting for close lock at 1676325454446Disabling compacts and flushes for region at 1676325454446Disabling writes for close at 1676325454447 (+1 ms)Writing region close event to WAL at 1676325454447Closed at 1676325454447 2023-02-13 21:57:34,452 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676325454277 execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 21:57:34,453 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325454453"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325454453"}]},"ts":"1676325454453"} 2023-02-13 21:57:34,453 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325454453"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325454453"}]},"ts":"1676325454453"} 2023-02-13 21:57:34,454 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325454453"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325454453"}]},"ts":"1676325454453"} 2023-02-13 21:57:34,454 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325454453"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325454453"}]},"ts":"1676325454453"} 2023-02-13 21:57:34,454 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325454453"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325454453"}]},"ts":"1676325454453"} 2023-02-13 21:57:34,454 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325454453"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325454453"}]},"ts":"1676325454453"} 2023-02-13 21:57:34,521 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-13 21:57:34,522 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676325454277 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 21:57:34,522 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325454522"}]},"ts":"1676325454522"} 2023-02-13 21:57:34,525 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676325454277, state=ENABLING in hbase:meta 2023-02-13 21:57:34,528 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-02-13 21:57:34,532 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 21:57:34,532 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 21:57:34,532 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 21:57:34,532 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 21:57:34,533 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, ASSIGN}, {pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, ASSIGN}, {pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, ASSIGN}, {pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, ASSIGN}, {pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, ASSIGN}, {pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, ASSIGN}] 2023-02-13 21:57:34,537 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, ASSIGN 2023-02-13 21:57:34,537 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, ASSIGN 2023-02-13 21:57:34,538 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, ASSIGN 2023-02-13 21:57:34,538 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, ASSIGN 2023-02-13 21:57:34,539 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,36339,1676325451407; forceNewPlan=false, retain=false 2023-02-13 21:57:34,540 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,45015,1676325451448; forceNewPlan=false, retain=false 2023-02-13 21:57:34,540 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, ASSIGN 2023-02-13 21:57:34,540 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,39523,1676325451480; forceNewPlan=false, retain=false 2023-02-13 21:57:34,540 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,39523,1676325451480; forceNewPlan=false, retain=false 2023-02-13 21:57:34,541 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, ASSIGN 2023-02-13 21:57:34,542 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,36339,1676325451407; forceNewPlan=false, retain=false 2023-02-13 21:57:34,542 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,45015,1676325451448; forceNewPlan=false, retain=false 2023-02-13 21:57:34,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 21:57:34,690 INFO [jenkins-hbase20:46557] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 21:57:34,697 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=4e757d660af6a5aaa6e2c43f8b91311d, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:34,697 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=992f98999ce7b2c13e6adb6eca28ff70, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:34,697 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=676fb98d6cc62f0fdbda4ce59aad85cb, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:34,697 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=1f6dc79bf11da8444a0870d779d2fb4d, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:34,698 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325454697"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325454697"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325454697"}]},"ts":"1676325454697"} 2023-02-13 21:57:34,697 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=44e821dce55cb29b05989bca574cff62, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:34,699 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325454697"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325454697"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325454697"}]},"ts":"1676325454697"} 2023-02-13 21:57:34,699 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325454697"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325454697"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325454697"}]},"ts":"1676325454697"} 2023-02-13 21:57:34,699 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325454697"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325454697"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325454697"}]},"ts":"1676325454697"} 2023-02-13 21:57:34,698 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325454697"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325454697"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325454697"}]},"ts":"1676325454697"} 2023-02-13 21:57:34,702 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=11, state=RUNNABLE; OpenRegionProcedure 992f98999ce7b2c13e6adb6eca28ff70, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:57:34,704 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=10, state=RUNNABLE; OpenRegionProcedure 1f6dc79bf11da8444a0870d779d2fb4d, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:57:34,705 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=13, state=RUNNABLE; OpenRegionProcedure 676fb98d6cc62f0fdbda4ce59aad85cb, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:57:34,707 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=12, state=RUNNABLE; OpenRegionProcedure 44e821dce55cb29b05989bca574cff62, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:57:34,709 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=14, state=RUNNABLE; OpenRegionProcedure 4e757d660af6a5aaa6e2c43f8b91311d, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:57:34,713 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=251bc68b009792a14e42934b238a6375, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:34,714 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325454713"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325454713"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325454713"}]},"ts":"1676325454713"} 2023-02-13 21:57:34,717 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=15, state=RUNNABLE; OpenRegionProcedure 251bc68b009792a14e42934b238a6375, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:57:34,861 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:34,861 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 21:57:34,865 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44958, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 21:57:34,871 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:34,872 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:34,873 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:34,873 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 992f98999ce7b2c13e6adb6eca28ff70, NAME => 'testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 21:57:34,873 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1f6dc79bf11da8444a0870d779d2fb4d, NAME => 'testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 21:57:34,873 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 44e821dce55cb29b05989bca574cff62, NAME => 'testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 21:57:34,873 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:34,873 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:34,873 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:34,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:34,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:34,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:34,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:34,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:34,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:34,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:34,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:34,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:34,877 INFO [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:34,877 INFO [StoreOpener-44e821dce55cb29b05989bca574cff62-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:34,877 INFO [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:34,880 DEBUG [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf 2023-02-13 21:57:34,880 DEBUG [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf 2023-02-13 21:57:34,880 DEBUG [StoreOpener-44e821dce55cb29b05989bca574cff62-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf 2023-02-13 21:57:34,880 DEBUG [StoreOpener-44e821dce55cb29b05989bca574cff62-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf 2023-02-13 21:57:34,880 DEBUG [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf 2023-02-13 21:57:34,880 DEBUG [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf 2023-02-13 21:57:34,883 INFO [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1f6dc79bf11da8444a0870d779d2fb4d columnFamilyName cf 2023-02-13 21:57:34,883 INFO [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 992f98999ce7b2c13e6adb6eca28ff70 columnFamilyName cf 2023-02-13 21:57:34,883 INFO [StoreOpener-44e821dce55cb29b05989bca574cff62-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 44e821dce55cb29b05989bca574cff62 columnFamilyName cf 2023-02-13 21:57:34,887 INFO [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] regionserver.HStore(310): Store=992f98999ce7b2c13e6adb6eca28ff70/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:34,887 INFO [StoreOpener-44e821dce55cb29b05989bca574cff62-1] regionserver.HStore(310): Store=44e821dce55cb29b05989bca574cff62/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:34,893 INFO [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] regionserver.HStore(310): Store=1f6dc79bf11da8444a0870d779d2fb4d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:34,896 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:34,896 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:34,896 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:34,897 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:34,897 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:34,897 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:34,901 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:34,901 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:34,901 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:34,906 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 21:57:34,907 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 44e821dce55cb29b05989bca574cff62; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72190541, jitterRate=0.07572288811206818}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:34,907 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 44e821dce55cb29b05989bca574cff62: Running coprocessor pre-open hook at 1676325454875Writing region info on filesystem at 1676325454875Initializing all the Stores at 1676325454876 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325454876Cleaning up temporary data from old regions at 1676325454897 (+21 ms)Cleaning up detritus from prior splits at 1676325454898 (+1 ms)Running coprocessor post-open hooks at 1676325454907 (+9 ms)Region opened successfully at 1676325454907 2023-02-13 21:57:34,910 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 21:57:34,911 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 992f98999ce7b2c13e6adb6eca28ff70; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60018025, jitterRate=-0.10566173493862152}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:34,911 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 992f98999ce7b2c13e6adb6eca28ff70: Running coprocessor pre-open hook at 1676325454875Writing region info on filesystem at 1676325454875Initializing all the Stores at 1676325454876 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325454876Cleaning up temporary data from old regions at 1676325454897 (+21 ms)Cleaning up detritus from prior splits at 1676325454898 (+1 ms)Running coprocessor post-open hooks at 1676325454911 (+13 ms)Region opened successfully at 1676325454911 2023-02-13 21:57:34,911 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62., pid=19, masterSystemTime=1676325454861 2023-02-13 21:57:34,916 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 21:57:34,920 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 1f6dc79bf11da8444a0870d779d2fb4d; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71872978, jitterRate=0.07099083065986633}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:34,920 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70., pid=16, masterSystemTime=1676325454860 2023-02-13 21:57:34,920 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 1f6dc79bf11da8444a0870d779d2fb4d: Running coprocessor pre-open hook at 1676325454874Writing region info on filesystem at 1676325454874Initializing all the Stores at 1676325454876 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325454876Cleaning up temporary data from old regions at 1676325454897 (+21 ms)Cleaning up detritus from prior splits at 1676325454898 (+1 ms)Running coprocessor post-open hooks at 1676325454920 (+22 ms)Region opened successfully at 1676325454920 2023-02-13 21:57:34,921 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d., pid=17, masterSystemTime=1676325454860 2023-02-13 21:57:34,921 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:34,922 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:34,922 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:34,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 4e757d660af6a5aaa6e2c43f8b91311d, NAME => 'testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 21:57:34,923 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=44e821dce55cb29b05989bca574cff62, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:34,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:34,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:34,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:34,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:34,924 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:34,924 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325454923"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325454923"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325454923"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325454923"}]},"ts":"1676325454923"} 2023-02-13 21:57:34,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:34,924 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:34,925 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 251bc68b009792a14e42934b238a6375, NAME => 'testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 21:57:34,925 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:34,925 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:34,925 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:34,925 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:34,925 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=992f98999ce7b2c13e6adb6eca28ff70, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:34,926 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:34,926 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:34,926 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325454925"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325454925"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325454925"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325454925"}]},"ts":"1676325454925"} 2023-02-13 21:57:34,926 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:34,926 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 676fb98d6cc62f0fdbda4ce59aad85cb, NAME => 'testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 21:57:34,926 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:34,926 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:34,927 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:34,927 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:34,927 INFO [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:34,927 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=1f6dc79bf11da8444a0870d779d2fb4d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:34,928 INFO [StoreOpener-251bc68b009792a14e42934b238a6375-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:34,928 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325454927"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325454927"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325454927"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325454927"}]},"ts":"1676325454927"} 2023-02-13 21:57:34,929 INFO [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:34,930 DEBUG [StoreOpener-251bc68b009792a14e42934b238a6375-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf 2023-02-13 21:57:34,930 DEBUG [StoreOpener-251bc68b009792a14e42934b238a6375-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf 2023-02-13 21:57:34,931 DEBUG [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf 2023-02-13 21:57:34,931 DEBUG [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf 2023-02-13 21:57:34,931 INFO [StoreOpener-251bc68b009792a14e42934b238a6375-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 251bc68b009792a14e42934b238a6375 columnFamilyName cf 2023-02-13 21:57:34,931 INFO [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4e757d660af6a5aaa6e2c43f8b91311d columnFamilyName cf 2023-02-13 21:57:34,933 DEBUG [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf 2023-02-13 21:57:34,935 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=12 2023-02-13 21:57:34,933 INFO [StoreOpener-251bc68b009792a14e42934b238a6375-1] regionserver.HStore(310): Store=251bc68b009792a14e42934b238a6375/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:34,936 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=12, state=SUCCESS; OpenRegionProcedure 44e821dce55cb29b05989bca574cff62, server=jenkins-hbase20.apache.org,45015,1676325451448 in 221 msec 2023-02-13 21:57:34,933 INFO [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] regionserver.HStore(310): Store=4e757d660af6a5aaa6e2c43f8b91311d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:34,935 DEBUG [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf 2023-02-13 21:57:34,937 INFO [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 676fb98d6cc62f0fdbda4ce59aad85cb columnFamilyName cf 2023-02-13 21:57:34,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375 2023-02-13 21:57:34,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375 2023-02-13 21:57:34,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:34,938 INFO [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] regionserver.HStore(310): Store=676fb98d6cc62f0fdbda4ce59aad85cb/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:34,940 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=11 2023-02-13 21:57:34,941 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=11, state=SUCCESS; OpenRegionProcedure 992f98999ce7b2c13e6adb6eca28ff70, server=jenkins-hbase20.apache.org,36339,1676325451407 in 230 msec 2023-02-13 21:57:34,941 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:34,943 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:34,944 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:34,944 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=10 2023-02-13 21:57:34,944 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=10, state=SUCCESS; OpenRegionProcedure 1f6dc79bf11da8444a0870d779d2fb4d, server=jenkins-hbase20.apache.org,39523,1676325451480 in 231 msec 2023-02-13 21:57:34,944 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, ASSIGN in 403 msec 2023-02-13 21:57:34,946 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, ASSIGN in 408 msec 2023-02-13 21:57:34,946 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:34,948 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, ASSIGN in 411 msec 2023-02-13 21:57:34,952 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:34,952 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:34,952 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 21:57:34,953 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 251bc68b009792a14e42934b238a6375; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63732399, jitterRate=-0.05031324923038483}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:34,953 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 251bc68b009792a14e42934b238a6375: Running coprocessor pre-open hook at 1676325454925Writing region info on filesystem at 1676325454925Initializing all the Stores at 1676325454927 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325454927Cleaning up temporary data from old regions at 1676325454940 (+13 ms)Cleaning up detritus from prior splits at 1676325454940Running coprocessor post-open hooks at 1676325454953 (+13 ms)Region opened successfully at 1676325454953 2023-02-13 21:57:34,954 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375., pid=21, masterSystemTime=1676325454860 2023-02-13 21:57:34,956 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 21:57:34,958 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 21:57:34,958 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:34,958 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:34,958 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 676fb98d6cc62f0fdbda4ce59aad85cb; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61746426, jitterRate=-0.07990655303001404}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:34,958 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 676fb98d6cc62f0fdbda4ce59aad85cb: Running coprocessor pre-open hook at 1676325454927Writing region info on filesystem at 1676325454927Initializing all the Stores at 1676325454928 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325454928Cleaning up temporary data from old regions at 1676325454945 (+17 ms)Cleaning up detritus from prior splits at 1676325454946 (+1 ms)Running coprocessor post-open hooks at 1676325454958 (+12 ms)Region opened successfully at 1676325454958 2023-02-13 21:57:34,959 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=251bc68b009792a14e42934b238a6375, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:34,959 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 4e757d660af6a5aaa6e2c43f8b91311d; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59073301, jitterRate=-0.11973921954631805}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:34,960 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb., pid=18, masterSystemTime=1676325454860 2023-02-13 21:57:34,960 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 4e757d660af6a5aaa6e2c43f8b91311d: Running coprocessor pre-open hook at 1676325454925Writing region info on filesystem at 1676325454925Initializing all the Stores at 1676325454926 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325454926Cleaning up temporary data from old regions at 1676325454944 (+18 ms)Cleaning up detritus from prior splits at 1676325454946 (+2 ms)Running coprocessor post-open hooks at 1676325454960 (+14 ms)Region opened successfully at 1676325454960 2023-02-13 21:57:34,960 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325454959"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325454959"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325454959"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325454959"}]},"ts":"1676325454959"} 2023-02-13 21:57:34,962 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d., pid=20, masterSystemTime=1676325454861 2023-02-13 21:57:34,967 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:34,969 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:34,970 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=676fb98d6cc62f0fdbda4ce59aad85cb, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:34,970 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:34,970 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:34,970 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325454970"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325454970"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325454970"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325454970"}]},"ts":"1676325454970"} 2023-02-13 21:57:34,973 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=4e757d660af6a5aaa6e2c43f8b91311d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:34,973 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325454972"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325454972"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325454972"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325454972"}]},"ts":"1676325454972"} 2023-02-13 21:57:34,974 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=15 2023-02-13 21:57:34,974 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=15, state=SUCCESS; OpenRegionProcedure 251bc68b009792a14e42934b238a6375, server=jenkins-hbase20.apache.org,36339,1676325451407 in 250 msec 2023-02-13 21:57:34,977 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, ASSIGN in 441 msec 2023-02-13 21:57:34,979 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=13 2023-02-13 21:57:34,979 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=13, state=SUCCESS; OpenRegionProcedure 676fb98d6cc62f0fdbda4ce59aad85cb, server=jenkins-hbase20.apache.org,39523,1676325451480 in 270 msec 2023-02-13 21:57:34,981 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, ASSIGN in 446 msec 2023-02-13 21:57:34,983 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=14 2023-02-13 21:57:34,983 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=14, state=SUCCESS; OpenRegionProcedure 4e757d660af6a5aaa6e2c43f8b91311d, server=jenkins-hbase20.apache.org,45015,1676325451448 in 269 msec 2023-02-13 21:57:34,987 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=9 2023-02-13 21:57:34,987 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, ASSIGN in 450 msec 2023-02-13 21:57:34,988 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676325454277 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 21:57:34,988 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325454988"}]},"ts":"1676325454988"} 2023-02-13 21:57:34,991 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676325454277, state=ENABLED in hbase:meta 2023-02-13 21:57:34,994 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676325454277 execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 21:57:34,997 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testRestoreSnapshot-1676325454277 in 691 msec 2023-02-13 21:57:35,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 21:57:35,088 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testRestoreSnapshot-1676325454277, procId: 9 completed 2023-02-13 21:57:35,094 DEBUG [Listener at localhost.localdomain/44075] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676325454277 2023-02-13 21:57:35,096 DEBUG [Listener at localhost.localdomain/44075] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:35,099 INFO [Listener at localhost.localdomain/44075] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 21:57:35,120 DEBUG [hconnection-0x1d9ca414-shared-pool-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 21:57:35,123 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55826, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 21:57:35,131 DEBUG [hconnection-0x1d9ca414-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 21:57:35,135 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44968, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 21:57:35,147 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676325454277 2023-02-13 21:57:35,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.HMaster$11(2405): Client=jenkins//148.251.75.209 disable testRestoreSnapshot-1676325454277 2023-02-13 21:57:35,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=22, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:35,161 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325455160"}]},"ts":"1676325455160"} 2023-02-13 21:57:35,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-13 21:57:35,163 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676325454277, state=DISABLING in hbase:meta 2023-02-13 21:57:35,164 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676325454277 to state=DISABLING 2023-02-13 21:57:35,167 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, UNASSIGN}, {pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, UNASSIGN}, {pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, UNASSIGN}, {pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, UNASSIGN}, {pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, UNASSIGN}, {pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, UNASSIGN}] 2023-02-13 21:57:35,170 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, UNASSIGN 2023-02-13 21:57:35,170 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, UNASSIGN 2023-02-13 21:57:35,170 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, UNASSIGN 2023-02-13 21:57:35,170 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, UNASSIGN 2023-02-13 21:57:35,170 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, UNASSIGN 2023-02-13 21:57:35,172 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=1f6dc79bf11da8444a0870d779d2fb4d, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:35,172 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=992f98999ce7b2c13e6adb6eca28ff70, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:35,172 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=44e821dce55cb29b05989bca574cff62, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:35,172 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=676fb98d6cc62f0fdbda4ce59aad85cb, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:35,172 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325455171"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325455171"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325455171"}]},"ts":"1676325455171"} 2023-02-13 21:57:35,172 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325455172"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325455172"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325455172"}]},"ts":"1676325455172"} 2023-02-13 21:57:35,172 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325455172"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325455172"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325455172"}]},"ts":"1676325455172"} 2023-02-13 21:57:35,172 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325455171"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325455171"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325455171"}]},"ts":"1676325455171"} 2023-02-13 21:57:35,173 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=251bc68b009792a14e42934b238a6375, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:35,173 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325455173"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325455173"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325455173"}]},"ts":"1676325455173"} 2023-02-13 21:57:35,175 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=24, state=RUNNABLE; CloseRegionProcedure 992f98999ce7b2c13e6adb6eca28ff70, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:57:35,176 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=25, state=RUNNABLE; CloseRegionProcedure 44e821dce55cb29b05989bca574cff62, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:57:35,178 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=31, ppid=26, state=RUNNABLE; CloseRegionProcedure 676fb98d6cc62f0fdbda4ce59aad85cb, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:57:35,179 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=23, state=RUNNABLE; CloseRegionProcedure 1f6dc79bf11da8444a0870d779d2fb4d, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:57:35,181 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=28, state=RUNNABLE; CloseRegionProcedure 251bc68b009792a14e42934b238a6375, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:57:35,185 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, UNASSIGN 2023-02-13 21:57:35,188 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=4e757d660af6a5aaa6e2c43f8b91311d, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:35,188 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325455188"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325455188"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325455188"}]},"ts":"1676325455188"} 2023-02-13 21:57:35,190 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=34, ppid=27, state=RUNNABLE; CloseRegionProcedure 4e757d660af6a5aaa6e2c43f8b91311d, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:57:35,339 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:35,339 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:35,339 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:35,340 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 992f98999ce7b2c13e6adb6eca28ff70, disabling compactions & flushes 2023-02-13 21:57:35,340 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 676fb98d6cc62f0fdbda4ce59aad85cb, disabling compactions & flushes 2023-02-13 21:57:35,340 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:35,340 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 44e821dce55cb29b05989bca574cff62, disabling compactions & flushes 2023-02-13 21:57:35,340 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:35,340 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:35,341 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. after waiting 0 ms 2023-02-13 21:57:35,341 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:35,341 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:35,341 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:35,341 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. after waiting 0 ms 2023-02-13 21:57:35,341 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:35,341 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:35,341 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. after waiting 0 ms 2023-02-13 21:57:35,342 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:35,352 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 21:57:35,352 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 21:57:35,353 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 21:57:35,353 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:35,353 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 992f98999ce7b2c13e6adb6eca28ff70: Waiting for close lock at 1676325455339Running coprocessor pre-close hooks at 1676325455339Disabling compacts and flushes for region at 1676325455339Disabling writes for close at 1676325455341 (+2 ms)Writing region close event to WAL at 1676325455341Running coprocessor post-close hooks at 1676325455353 (+12 ms)Closed at 1676325455353 2023-02-13 21:57:35,353 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:35,353 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 676fb98d6cc62f0fdbda4ce59aad85cb: Waiting for close lock at 1676325455339Running coprocessor pre-close hooks at 1676325455339Disabling compacts and flushes for region at 1676325455339Disabling writes for close at 1676325455341 (+2 ms)Writing region close event to WAL at 1676325455343 (+2 ms)Running coprocessor post-close hooks at 1676325455353 (+10 ms)Closed at 1676325455353 2023-02-13 21:57:35,354 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:35,354 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 44e821dce55cb29b05989bca574cff62: Waiting for close lock at 1676325455340Running coprocessor pre-close hooks at 1676325455340Disabling compacts and flushes for region at 1676325455340Disabling writes for close at 1676325455342 (+2 ms)Writing region close event to WAL at 1676325455344 (+2 ms)Running coprocessor post-close hooks at 1676325455353 (+9 ms)Closed at 1676325455354 (+1 ms) 2023-02-13 21:57:35,356 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:35,356 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:35,357 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 251bc68b009792a14e42934b238a6375, disabling compactions & flushes 2023-02-13 21:57:35,357 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:35,357 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:35,357 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. after waiting 0 ms 2023-02-13 21:57:35,357 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:35,358 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=992f98999ce7b2c13e6adb6eca28ff70, regionState=CLOSED 2023-02-13 21:57:35,358 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325455357"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325455357"}]},"ts":"1676325455357"} 2023-02-13 21:57:35,358 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:35,358 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:35,358 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 1f6dc79bf11da8444a0870d779d2fb4d, disabling compactions & flushes 2023-02-13 21:57:35,359 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:35,359 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:35,359 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. after waiting 0 ms 2023-02-13 21:57:35,359 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:35,359 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=676fb98d6cc62f0fdbda4ce59aad85cb, regionState=CLOSED 2023-02-13 21:57:35,361 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:35,362 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325455359"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325455359"}]},"ts":"1676325455359"} 2023-02-13 21:57:35,362 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:35,363 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 4e757d660af6a5aaa6e2c43f8b91311d, disabling compactions & flushes 2023-02-13 21:57:35,363 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:35,363 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:35,363 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. after waiting 0 ms 2023-02-13 21:57:35,363 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:35,363 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=44e821dce55cb29b05989bca574cff62, regionState=CLOSED 2023-02-13 21:57:35,364 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325455363"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325455363"}]},"ts":"1676325455363"} 2023-02-13 21:57:35,368 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 21:57:35,368 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=24 2023-02-13 21:57:35,369 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=24, state=SUCCESS; CloseRegionProcedure 992f98999ce7b2c13e6adb6eca28ff70, server=jenkins-hbase20.apache.org,36339,1676325451407 in 188 msec 2023-02-13 21:57:35,369 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:35,369 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 251bc68b009792a14e42934b238a6375: Waiting for close lock at 1676325455356Running coprocessor pre-close hooks at 1676325455356Disabling compacts and flushes for region at 1676325455356Disabling writes for close at 1676325455357 (+1 ms)Writing region close event to WAL at 1676325455357Running coprocessor post-close hooks at 1676325455369 (+12 ms)Closed at 1676325455369 2023-02-13 21:57:35,371 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 21:57:35,371 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=31, resume processing ppid=26 2023-02-13 21:57:35,372 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=26, state=SUCCESS; CloseRegionProcedure 676fb98d6cc62f0fdbda4ce59aad85cb, server=jenkins-hbase20.apache.org,39523,1676325451480 in 187 msec 2023-02-13 21:57:35,373 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=25 2023-02-13 21:57:35,374 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=25, state=SUCCESS; CloseRegionProcedure 44e821dce55cb29b05989bca574cff62, server=jenkins-hbase20.apache.org,45015,1676325451448 in 191 msec 2023-02-13 21:57:35,374 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, UNASSIGN in 203 msec 2023-02-13 21:57:35,374 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:35,375 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=251bc68b009792a14e42934b238a6375, regionState=CLOSED 2023-02-13 21:57:35,376 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, UNASSIGN in 206 msec 2023-02-13 21:57:35,376 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325455375"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325455375"}]},"ts":"1676325455375"} 2023-02-13 21:57:35,376 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, UNASSIGN in 208 msec 2023-02-13 21:57:35,380 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:35,380 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 4e757d660af6a5aaa6e2c43f8b91311d: Waiting for close lock at 1676325455362Running coprocessor pre-close hooks at 1676325455362Disabling compacts and flushes for region at 1676325455362Disabling writes for close at 1676325455363 (+1 ms)Writing region close event to WAL at 1676325455364 (+1 ms)Running coprocessor post-close hooks at 1676325455380 (+16 ms)Closed at 1676325455380 2023-02-13 21:57:35,383 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=28 2023-02-13 21:57:35,383 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=28, state=SUCCESS; CloseRegionProcedure 251bc68b009792a14e42934b238a6375, server=jenkins-hbase20.apache.org,36339,1676325451407 in 197 msec 2023-02-13 21:57:35,383 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:35,384 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=4e757d660af6a5aaa6e2c43f8b91311d, regionState=CLOSED 2023-02-13 21:57:35,385 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325455384"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325455384"}]},"ts":"1676325455384"} 2023-02-13 21:57:35,387 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, UNASSIGN in 217 msec 2023-02-13 21:57:35,392 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 21:57:35,393 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:35,393 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 1f6dc79bf11da8444a0870d779d2fb4d: Waiting for close lock at 1676325455358Running coprocessor pre-close hooks at 1676325455358Disabling compacts and flushes for region at 1676325455358Disabling writes for close at 1676325455359 (+1 ms)Writing region close event to WAL at 1676325455359Running coprocessor post-close hooks at 1676325455393 (+34 ms)Closed at 1676325455393 2023-02-13 21:57:35,393 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=34, resume processing ppid=27 2023-02-13 21:57:35,393 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=34, ppid=27, state=SUCCESS; CloseRegionProcedure 4e757d660af6a5aaa6e2c43f8b91311d, server=jenkins-hbase20.apache.org,45015,1676325451448 in 197 msec 2023-02-13 21:57:35,395 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, UNASSIGN in 227 msec 2023-02-13 21:57:35,395 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:35,396 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=1f6dc79bf11da8444a0870d779d2fb4d, regionState=CLOSED 2023-02-13 21:57:35,396 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325455396"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325455396"}]},"ts":"1676325455396"} 2023-02-13 21:57:35,401 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=23 2023-02-13 21:57:35,401 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=23, state=SUCCESS; CloseRegionProcedure 1f6dc79bf11da8444a0870d779d2fb4d, server=jenkins-hbase20.apache.org,39523,1676325451480 in 219 msec 2023-02-13 21:57:35,403 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-02-13 21:57:35,403 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, UNASSIGN in 235 msec 2023-02-13 21:57:35,404 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325455404"}]},"ts":"1676325455404"} 2023-02-13 21:57:35,406 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676325454277, state=DISABLED in hbase:meta 2023-02-13 21:57:35,407 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676325454277 to state=DISABLED 2023-02-13 21:57:35,410 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=22, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676325454277 in 256 msec 2023-02-13 21:57:35,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-13 21:57:35,413 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676325454277, procId: 22 completed 2023-02-13 21:57:35,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1580): Client=jenkins//148.251.75.209 snapshot request for:{ ss=emptySnaptb-1676325454277 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 } 2023-02-13 21:57:35,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676325455421 (current time:1676325455421). 2023-02-13 21:57:35,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 21:57:35,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] zookeeper.ReadOnlyZKClient(139): Connect 0x1fcafe71 to 127.0.0.1:58580 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 21:57:35,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@103f26a7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 21:57:35,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 21:57:35,432 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34548, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 21:57:35,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1fcafe71 to 127.0.0.1:58580 2023-02-13 21:57:35,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 21:57:35,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 21:57:35,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1676325454277 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 } 2023-02-13 21:57:35,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676325454277, type=EXCLUSIVE 2023-02-13 21:57:35,455 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676325454277, type=EXCLUSIVE 2023-02-13 21:57:35,457 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=35, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676325454277, type=EXCLUSIVE; timeout=600000, timestamp=1676326055457 2023-02-13 21:57:35,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1676325454277 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 } 2023-02-13 21:57:35,457 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1676325454277 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676325454277 2023-02-13 21:57:35,462 DEBUG [Listener at localhost.localdomain/44075] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1676325454277 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 21:57:35,462 DEBUG [Listener at localhost.localdomain/44075] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 21:57:35,476 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676325454277 table=testRestoreSnapshot-1676325454277 type=DISABLED ttl=0 } 2023-02-13 21:57:35,479 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:35,479 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:35,480 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:35,480 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:35,480 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:35,480 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:35,480 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:35,484 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:35,484 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:35,484 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:35,484 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:35,484 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:35,484 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:35,484 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:35,493 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 21:57:35,493 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 21:57:35,493 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 21:57:35,493 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 21:57:35,493 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 21:57:35,495 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 21:57:35,495 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 21:57:35,712 DEBUG [Listener at localhost.localdomain/44075] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 21:57:35,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676325454277 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 } is done 2023-02-13 21:57:35,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(404): Snapshoting '{ ss=emptySnaptb-1676325454277 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 }' is still in progress! 2023-02-13 21:57:35,716 DEBUG [Listener at localhost.localdomain/44075] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-13 21:57:35,972 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1676325454277 table=testRestoreSnapshot-1676325454277 type=DISABLED ttl=0 } as finished. 2023-02-13 21:57:35,973 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1676325454277 2023-02-13 21:57:35,975 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/emptySnaptb-1676325454277 2023-02-13 21:57:36,030 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/emptySnaptb-1676325454277 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/emptySnaptb-1676325454277 2023-02-13 21:57:36,081 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1676325454277 of table testRestoreSnapshot-1676325454277 completed 2023-02-13 21:57:36,081 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/emptySnaptb-1676325454277 2023-02-13 21:57:36,082 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/emptySnaptb-1676325454277 2023-02-13 21:57:36,082 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1676325454277 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676325454277 at 1676325455457Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676325454277 table=testRestoreSnapshot-1676325454277 type=DISABLED ttl=0 } at 1676325455476 (+19 ms)Consolidate snapshot: emptySnaptb-1676325454277 at 1676325455973 (+497 ms)Loading Region manifests for emptySnaptb-1676325454277 at 1676325455973Writing data manifest for emptySnaptb-1676325454277 at 1676325455989 (+16 ms)Verifying snapshot: emptySnaptb-1676325454277 at 1676325456013 (+24 ms)Snapshot emptySnaptb-1676325454277 of table testRestoreSnapshot-1676325454277 completed at 1676325456081 (+68 ms) 2023-02-13 21:57:36,084 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=35, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676325454277, type=EXCLUSIVE 2023-02-13 21:57:36,087 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676325454277, type=EXCLUSIVE in 637 msec 2023-02-13 21:57:36,216 DEBUG [Listener at localhost.localdomain/44075] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 21:57:36,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676325454277 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 } is done 2023-02-13 21:57:36,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1676325454277 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 21:57:36,221 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676325454277 2023-02-13 21:57:36,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.HMaster$10(2372): Client=jenkins//148.251.75.209 enable testRestoreSnapshot-1676325454277 2023-02-13 21:57:36,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=36, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:36,242 INFO [PEWorker-1] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676325454277 2023-02-13 21:57:36,242 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325456242"}]},"ts":"1676325456242"} 2023-02-13 21:57:36,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-13 21:57:36,245 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676325454277, state=ENABLING in hbase:meta 2023-02-13 21:57:36,249 DEBUG [PEWorker-1] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 21:57:36,254 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, ASSIGN}, {pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, ASSIGN}, {pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, ASSIGN}, {pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, ASSIGN}, {pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, ASSIGN}, {pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, ASSIGN}] 2023-02-13 21:57:36,258 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, ASSIGN 2023-02-13 21:57:36,260 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:36,260 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,39523,1676325451480; forceNewPlan=false, retain=true 2023-02-13 21:57:36,262 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, ASSIGN 2023-02-13 21:57:36,265 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:36,265 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,45015,1676325451448; forceNewPlan=false, retain=true 2023-02-13 21:57:36,267 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, ASSIGN 2023-02-13 21:57:36,269 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:36,269 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,36339,1676325451407; forceNewPlan=false, retain=true 2023-02-13 21:57:36,271 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, ASSIGN 2023-02-13 21:57:36,273 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, ASSIGN 2023-02-13 21:57:36,274 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, ASSIGN 2023-02-13 21:57:36,276 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:36,276 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,39523,1676325451480; forceNewPlan=false, retain=true 2023-02-13 21:57:36,278 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:36,278 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,45015,1676325451448; forceNewPlan=false, retain=true 2023-02-13 21:57:36,279 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:36,279 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,36339,1676325451407; forceNewPlan=false, retain=true 2023-02-13 21:57:36,411 INFO [jenkins-hbase20:46557] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 21:57:36,411 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=1f6dc79bf11da8444a0870d779d2fb4d, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:36,411 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=4e757d660af6a5aaa6e2c43f8b91311d, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:36,411 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=992f98999ce7b2c13e6adb6eca28ff70, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:36,412 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325456411"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325456411"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325456411"}]},"ts":"1676325456411"} 2023-02-13 21:57:36,411 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=44e821dce55cb29b05989bca574cff62, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:36,411 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=676fb98d6cc62f0fdbda4ce59aad85cb, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:36,412 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325456411"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325456411"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325456411"}]},"ts":"1676325456411"} 2023-02-13 21:57:36,412 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325456411"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325456411"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325456411"}]},"ts":"1676325456411"} 2023-02-13 21:57:36,412 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325456411"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325456411"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325456411"}]},"ts":"1676325456411"} 2023-02-13 21:57:36,412 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325456411"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325456411"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325456411"}]},"ts":"1676325456411"} 2023-02-13 21:57:36,414 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=43, ppid=37, state=RUNNABLE; OpenRegionProcedure 1f6dc79bf11da8444a0870d779d2fb4d, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:57:36,415 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=38, state=RUNNABLE; OpenRegionProcedure 992f98999ce7b2c13e6adb6eca28ff70, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:57:36,417 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=41, state=RUNNABLE; OpenRegionProcedure 4e757d660af6a5aaa6e2c43f8b91311d, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:57:36,418 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=46, ppid=40, state=RUNNABLE; OpenRegionProcedure 676fb98d6cc62f0fdbda4ce59aad85cb, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:57:36,420 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=47, ppid=39, state=RUNNABLE; OpenRegionProcedure 44e821dce55cb29b05989bca574cff62, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:57:36,420 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=251bc68b009792a14e42934b238a6375, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:36,420 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325456420"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325456420"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325456420"}]},"ts":"1676325456420"} 2023-02-13 21:57:36,423 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=42, state=RUNNABLE; OpenRegionProcedure 251bc68b009792a14e42934b238a6375, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:57:36,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-13 21:57:36,572 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:36,572 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 676fb98d6cc62f0fdbda4ce59aad85cb, NAME => 'testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 21:57:36,572 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:36,572 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:36,572 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:36,572 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:36,573 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:36,574 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 251bc68b009792a14e42934b238a6375, NAME => 'testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 21:57:36,574 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:36,574 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:36,574 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:36,574 INFO [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:36,574 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:36,575 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:36,575 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 4e757d660af6a5aaa6e2c43f8b91311d, NAME => 'testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 21:57:36,575 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:36,576 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:36,576 DEBUG [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf 2023-02-13 21:57:36,576 DEBUG [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf 2023-02-13 21:57:36,576 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:36,576 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:36,576 INFO [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 676fb98d6cc62f0fdbda4ce59aad85cb columnFamilyName cf 2023-02-13 21:57:36,576 INFO [StoreOpener-251bc68b009792a14e42934b238a6375-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:36,578 INFO [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] regionserver.HStore(310): Store=676fb98d6cc62f0fdbda4ce59aad85cb/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:36,578 INFO [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:36,579 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:36,579 DEBUG [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf 2023-02-13 21:57:36,580 DEBUG [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf 2023-02-13 21:57:36,580 DEBUG [StoreOpener-251bc68b009792a14e42934b238a6375-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf 2023-02-13 21:57:36,580 DEBUG [StoreOpener-251bc68b009792a14e42934b238a6375-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf 2023-02-13 21:57:36,580 INFO [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4e757d660af6a5aaa6e2c43f8b91311d columnFamilyName cf 2023-02-13 21:57:36,581 INFO [StoreOpener-251bc68b009792a14e42934b238a6375-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 251bc68b009792a14e42934b238a6375 columnFamilyName cf 2023-02-13 21:57:36,581 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:36,582 INFO [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] regionserver.HStore(310): Store=4e757d660af6a5aaa6e2c43f8b91311d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:36,583 INFO [StoreOpener-251bc68b009792a14e42934b238a6375-1] regionserver.HStore(310): Store=251bc68b009792a14e42934b238a6375/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:36,584 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:36,585 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375 2023-02-13 21:57:36,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:36,587 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375 2023-02-13 21:57:36,587 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:36,589 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 676fb98d6cc62f0fdbda4ce59aad85cb; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59776253, jitterRate=-0.10926441848278046}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:36,589 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 676fb98d6cc62f0fdbda4ce59aad85cb: Running coprocessor pre-open hook at 1676325456572Writing region info on filesystem at 1676325456572Initializing all the Stores at 1676325456574 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325456574Cleaning up temporary data from old regions at 1676325456583 (+9 ms)Cleaning up detritus from prior splits at 1676325456584 (+1 ms)Running coprocessor post-open hooks at 1676325456589 (+5 ms)Region opened successfully at 1676325456589 2023-02-13 21:57:36,592 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:36,592 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb., pid=46, masterSystemTime=1676325456567 2023-02-13 21:57:36,592 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:36,594 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 251bc68b009792a14e42934b238a6375; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73200195, jitterRate=0.09076790511608124}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:36,594 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 251bc68b009792a14e42934b238a6375: Running coprocessor pre-open hook at 1676325456574Writing region info on filesystem at 1676325456575 (+1 ms)Initializing all the Stores at 1676325456576 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325456576Cleaning up temporary data from old regions at 1676325456588 (+12 ms)Cleaning up detritus from prior splits at 1676325456588Running coprocessor post-open hooks at 1676325456594 (+6 ms)Region opened successfully at 1676325456594 2023-02-13 21:57:36,594 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 4e757d660af6a5aaa6e2c43f8b91311d; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62693600, jitterRate=-0.06579256057739258}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:36,594 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 4e757d660af6a5aaa6e2c43f8b91311d: Running coprocessor pre-open hook at 1676325456576Writing region info on filesystem at 1676325456576Initializing all the Stores at 1676325456577 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325456577Cleaning up temporary data from old regions at 1676325456587 (+10 ms)Cleaning up detritus from prior splits at 1676325456588 (+1 ms)Running coprocessor post-open hooks at 1676325456594 (+6 ms)Region opened successfully at 1676325456594 2023-02-13 21:57:36,595 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:36,595 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:36,595 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:36,595 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1f6dc79bf11da8444a0870d779d2fb4d, NAME => 'testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 21:57:36,595 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375., pid=48, masterSystemTime=1676325456569 2023-02-13 21:57:36,596 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d., pid=45, masterSystemTime=1676325456571 2023-02-13 21:57:36,596 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=676fb98d6cc62f0fdbda4ce59aad85cb, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:36,596 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325456596"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325456596"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325456596"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325456596"}]},"ts":"1676325456596"} 2023-02-13 21:57:36,596 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:36,596 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:36,596 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:36,596 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:36,598 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:36,598 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:36,598 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:36,598 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 992f98999ce7b2c13e6adb6eca28ff70, NAME => 'testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 21:57:36,599 INFO [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:36,599 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:36,599 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:36,599 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:36,599 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:36,600 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=251bc68b009792a14e42934b238a6375, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:36,600 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325456600"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325456600"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325456600"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325456600"}]},"ts":"1676325456600"} 2023-02-13 21:57:36,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:36,600 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:36,600 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:36,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 44e821dce55cb29b05989bca574cff62, NAME => 'testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 21:57:36,601 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:36,601 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:36,601 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=4e757d660af6a5aaa6e2c43f8b91311d, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:36,601 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:36,601 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:36,601 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325456601"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325456601"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325456601"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325456601"}]},"ts":"1676325456601"} 2023-02-13 21:57:36,602 DEBUG [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf 2023-02-13 21:57:36,602 DEBUG [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf 2023-02-13 21:57:36,603 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=46, resume processing ppid=40 2023-02-13 21:57:36,603 INFO [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1f6dc79bf11da8444a0870d779d2fb4d columnFamilyName cf 2023-02-13 21:57:36,603 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=46, ppid=40, state=SUCCESS; OpenRegionProcedure 676fb98d6cc62f0fdbda4ce59aad85cb, server=jenkins-hbase20.apache.org,39523,1676325451480 in 180 msec 2023-02-13 21:57:36,608 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, ASSIGN in 354 msec 2023-02-13 21:57:36,608 INFO [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] regionserver.HStore(310): Store=1f6dc79bf11da8444a0870d779d2fb4d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:36,609 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=42 2023-02-13 21:57:36,609 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=42, state=SUCCESS; OpenRegionProcedure 251bc68b009792a14e42934b238a6375, server=jenkins-hbase20.apache.org,36339,1676325451407 in 182 msec 2023-02-13 21:57:36,611 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=41 2023-02-13 21:57:36,611 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=41, state=SUCCESS; OpenRegionProcedure 4e757d660af6a5aaa6e2c43f8b91311d, server=jenkins-hbase20.apache.org,45015,1676325451448 in 190 msec 2023-02-13 21:57:36,612 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, ASSIGN in 360 msec 2023-02-13 21:57:36,612 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, ASSIGN in 362 msec 2023-02-13 21:57:36,615 INFO [StoreOpener-44e821dce55cb29b05989bca574cff62-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:36,615 INFO [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:36,616 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:36,617 DEBUG [StoreOpener-44e821dce55cb29b05989bca574cff62-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf 2023-02-13 21:57:36,617 DEBUG [StoreOpener-44e821dce55cb29b05989bca574cff62-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf 2023-02-13 21:57:36,618 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:36,618 INFO [StoreOpener-44e821dce55cb29b05989bca574cff62-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 44e821dce55cb29b05989bca574cff62 columnFamilyName cf 2023-02-13 21:57:36,619 DEBUG [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf 2023-02-13 21:57:36,619 DEBUG [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf 2023-02-13 21:57:36,619 INFO [StoreOpener-44e821dce55cb29b05989bca574cff62-1] regionserver.HStore(310): Store=44e821dce55cb29b05989bca574cff62/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:36,620 INFO [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 992f98999ce7b2c13e6adb6eca28ff70 columnFamilyName cf 2023-02-13 21:57:36,622 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:36,621 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:36,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:36,632 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 1f6dc79bf11da8444a0870d779d2fb4d; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60316204, jitterRate=-0.10121852159500122}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:36,633 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 1f6dc79bf11da8444a0870d779d2fb4d: Running coprocessor pre-open hook at 1676325456597Writing region info on filesystem at 1676325456597Initializing all the Stores at 1676325456598 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325456598Cleaning up temporary data from old regions at 1676325456618 (+20 ms)Cleaning up detritus from prior splits at 1676325456619 (+1 ms)Running coprocessor post-open hooks at 1676325456633 (+14 ms)Region opened successfully at 1676325456633 2023-02-13 21:57:36,634 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d., pid=43, masterSystemTime=1676325456567 2023-02-13 21:57:36,634 INFO [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] regionserver.HStore(310): Store=992f98999ce7b2c13e6adb6eca28ff70/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:36,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:36,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:36,638 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:36,639 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=1f6dc79bf11da8444a0870d779d2fb4d, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:36,639 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325456639"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325456639"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325456639"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325456639"}]},"ts":"1676325456639"} 2023-02-13 21:57:36,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:36,641 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:36,642 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 44e821dce55cb29b05989bca574cff62; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62411133, jitterRate=-0.07000164687633514}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:36,642 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 44e821dce55cb29b05989bca574cff62: Running coprocessor pre-open hook at 1676325456601Writing region info on filesystem at 1676325456601Initializing all the Stores at 1676325456603 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325456604 (+1 ms)Cleaning up temporary data from old regions at 1676325456633 (+29 ms)Cleaning up detritus from prior splits at 1676325456634 (+1 ms)Running coprocessor post-open hooks at 1676325456642 (+8 ms)Region opened successfully at 1676325456642 2023-02-13 21:57:36,644 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62., pid=47, masterSystemTime=1676325456571 2023-02-13 21:57:36,649 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=43, resume processing ppid=37 2023-02-13 21:57:36,650 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=43, ppid=37, state=SUCCESS; OpenRegionProcedure 1f6dc79bf11da8444a0870d779d2fb4d, server=jenkins-hbase20.apache.org,39523,1676325451480 in 228 msec 2023-02-13 21:57:36,651 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:36,651 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:36,658 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:36,659 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=44e821dce55cb29b05989bca574cff62, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:36,660 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325456659"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325456659"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325456659"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325456659"}]},"ts":"1676325456659"} 2023-02-13 21:57:36,663 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, ASSIGN in 401 msec 2023-02-13 21:57:36,664 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 992f98999ce7b2c13e6adb6eca28ff70; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65601556, jitterRate=-0.022460639476776123}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:36,664 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 992f98999ce7b2c13e6adb6eca28ff70: Running coprocessor pre-open hook at 1676325456599Writing region info on filesystem at 1676325456599Initializing all the Stores at 1676325456605 (+6 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325456605Cleaning up temporary data from old regions at 1676325456642 (+37 ms)Cleaning up detritus from prior splits at 1676325456643 (+1 ms)Running coprocessor post-open hooks at 1676325456664 (+21 ms)Region opened successfully at 1676325456664 2023-02-13 21:57:36,665 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70., pid=44, masterSystemTime=1676325456569 2023-02-13 21:57:36,668 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:36,669 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:36,669 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=47, resume processing ppid=39 2023-02-13 21:57:36,669 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=47, ppid=39, state=SUCCESS; OpenRegionProcedure 44e821dce55cb29b05989bca574cff62, server=jenkins-hbase20.apache.org,45015,1676325451448 in 244 msec 2023-02-13 21:57:36,670 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=992f98999ce7b2c13e6adb6eca28ff70, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:36,670 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325456670"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325456670"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325456670"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325456670"}]},"ts":"1676325456670"} 2023-02-13 21:57:36,674 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, ASSIGN in 420 msec 2023-02-13 21:57:36,677 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=38 2023-02-13 21:57:36,677 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=38, state=SUCCESS; OpenRegionProcedure 992f98999ce7b2c13e6adb6eca28ff70, server=jenkins-hbase20.apache.org,36339,1676325451407 in 258 msec 2023-02-13 21:57:36,680 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=38, resume processing ppid=36 2023-02-13 21:57:36,680 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=38, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, ASSIGN in 428 msec 2023-02-13 21:57:36,682 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325456682"}]},"ts":"1676325456682"} 2023-02-13 21:57:36,684 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676325454277, state=ENABLED in hbase:meta 2023-02-13 21:57:36,686 INFO [PEWorker-1] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676325454277' was successfully enabled. 2023-02-13 21:57:36,695 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676325454277 in 459 msec 2023-02-13 21:57:36,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-13 21:57:37,000 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676325454277, procId: 36 completed 2023-02-13 21:57:37,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 21:57:37,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 21:57:37,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 21:57:37,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 21:57:37,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 21:57:37,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 21:57:37,059 DEBUG [Listener at localhost.localdomain/44075] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676325454277 2023-02-13 21:57:37,059 DEBUG [Listener at localhost.localdomain/44075] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:37,061 INFO [Listener at localhost.localdomain/44075] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 21:57:37,081 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 21:57:37,083 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 21:57:37,085 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 21:57:37,087 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 21:57:37,089 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 21:57:37,098 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676325454277 2023-02-13 21:57:37,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.HMaster$11(2405): Client=jenkins//148.251.75.209 disable testRestoreSnapshot-1676325454277 2023-02-13 21:57:37,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=49, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:37,105 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325457105"}]},"ts":"1676325457105"} 2023-02-13 21:57:37,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-13 21:57:37,107 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676325454277, state=DISABLING in hbase:meta 2023-02-13 21:57:37,109 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676325454277 to state=DISABLING 2023-02-13 21:57:37,110 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, UNASSIGN}, {pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, UNASSIGN}, {pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, UNASSIGN}, {pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, UNASSIGN}, {pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, UNASSIGN}, {pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, UNASSIGN}] 2023-02-13 21:57:37,112 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, UNASSIGN 2023-02-13 21:57:37,112 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, UNASSIGN 2023-02-13 21:57:37,113 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, UNASSIGN 2023-02-13 21:57:37,113 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, UNASSIGN 2023-02-13 21:57:37,114 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, UNASSIGN 2023-02-13 21:57:37,115 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=992f98999ce7b2c13e6adb6eca28ff70, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:37,115 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325457115"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325457115"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325457115"}]},"ts":"1676325457115"} 2023-02-13 21:57:37,115 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=44e821dce55cb29b05989bca574cff62, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:37,116 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325457115"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325457115"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325457115"}]},"ts":"1676325457115"} 2023-02-13 21:57:37,116 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=1f6dc79bf11da8444a0870d779d2fb4d, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:37,116 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325457116"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325457116"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325457116"}]},"ts":"1676325457116"} 2023-02-13 21:57:37,118 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=51, state=RUNNABLE; CloseRegionProcedure 992f98999ce7b2c13e6adb6eca28ff70, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:57:37,119 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=251bc68b009792a14e42934b238a6375, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:37,119 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=4e757d660af6a5aaa6e2c43f8b91311d, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:37,120 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325457119"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325457119"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325457119"}]},"ts":"1676325457119"} 2023-02-13 21:57:37,120 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325457119"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325457119"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325457119"}]},"ts":"1676325457119"} 2023-02-13 21:57:37,121 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=57, ppid=52, state=RUNNABLE; CloseRegionProcedure 44e821dce55cb29b05989bca574cff62, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:57:37,122 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=50, state=RUNNABLE; CloseRegionProcedure 1f6dc79bf11da8444a0870d779d2fb4d, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:57:37,123 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=55, state=RUNNABLE; CloseRegionProcedure 251bc68b009792a14e42934b238a6375, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:57:37,125 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=54, state=RUNNABLE; CloseRegionProcedure 4e757d660af6a5aaa6e2c43f8b91311d, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:57:37,126 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, UNASSIGN 2023-02-13 21:57:37,136 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=676fb98d6cc62f0fdbda4ce59aad85cb, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:37,136 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325457136"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325457136"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325457136"}]},"ts":"1676325457136"} 2023-02-13 21:57:37,140 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=53, state=RUNNABLE; CloseRegionProcedure 676fb98d6cc62f0fdbda4ce59aad85cb, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:57:37,274 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:37,275 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 44e821dce55cb29b05989bca574cff62, disabling compactions & flushes 2023-02-13 21:57:37,275 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:37,275 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:37,275 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. after waiting 0 ms 2023-02-13 21:57:37,275 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:37,275 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:37,276 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 992f98999ce7b2c13e6adb6eca28ff70, disabling compactions & flushes 2023-02-13 21:57:37,276 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:37,277 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing 44e821dce55cb29b05989bca574cff62 1/1 column families, dataSize=2.09 KB heapSize=4.75 KB 2023-02-13 21:57:37,277 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:37,277 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. after waiting 0 ms 2023-02-13 21:57:37,277 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:37,277 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing 992f98999ce7b2c13e6adb6eca28ff70 1/1 column families, dataSize=1.76 KB heapSize=4.05 KB 2023-02-13 21:57:37,277 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:37,277 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 1f6dc79bf11da8444a0870d779d2fb4d, disabling compactions & flushes 2023-02-13 21:57:37,277 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:37,277 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:37,277 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. after waiting 0 ms 2023-02-13 21:57:37,277 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:37,278 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing 1f6dc79bf11da8444a0870d779d2fb4d 1/1 column families, dataSize=2.09 KB heapSize=4.75 KB 2023-02-13 21:57:37,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-13 21:57:37,799 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 2023-02-13 21:57:37,799 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 2023-02-13 21:57:37,799 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e 2023-02-13 21:57:37,855 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/.tmp/cf/ebd1b06b1c10407b903431c38f993f0f 2023-02-13 21:57:37,856 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/.tmp/cf/8795edd282b341fba51b1c197d77b4f9 2023-02-13 21:57:37,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-13 21:57:37,873 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/.tmp/cf/8795edd282b341fba51b1c197d77b4f9 as hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/8795edd282b341fba51b1c197d77b4f9 2023-02-13 21:57:37,876 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/.tmp/cf/ebd1b06b1c10407b903431c38f993f0f as hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/ebd1b06b1c10407b903431c38f993f0f 2023-02-13 21:57:37,889 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/8795edd282b341fba51b1c197d77b4f9, entries=32, sequenceid=8, filesize=10.4 K 2023-02-13 21:57:37,896 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/ebd1b06b1c10407b903431c38f993f0f, entries=27, sequenceid=8, filesize=9.5 K 2023-02-13 21:57:37,897 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for 44e821dce55cb29b05989bca574cff62 in 621ms, sequenceid=8, compaction requested=false 2023-02-13 21:57:37,898 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for 992f98999ce7b2c13e6adb6eca28ff70 in 621ms, sequenceid=8, compaction requested=false 2023-02-13 21:57:37,899 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676325454277' 2023-02-13 21:57:37,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676325454277' 2023-02-13 21:57:37,932 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/.tmp/cf/3d85d09beeee48a5b67b39f0abb4165e 2023-02-13 21:57:37,936 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 21:57:37,936 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 21:57:37,937 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:37,937 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:37,937 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 44e821dce55cb29b05989bca574cff62: Waiting for close lock at 1676325457274Running coprocessor pre-close hooks at 1676325457274Disabling compacts and flushes for region at 1676325457274Disabling writes for close at 1676325457275 (+1 ms)Obtaining lock to block concurrent updates at 1676325457277 (+2 ms)Preparing flush snapshotting stores in 44e821dce55cb29b05989bca574cff62 at 1676325457277Finished memstore snapshotting testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62., syncing WAL and waiting on mvcc, flushsize=dataSize=2142, getHeapSize=4848, getOffHeapSize=0, getCellsCount=32 at 1676325457285 (+8 ms)Flushing stores of testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. at 1676325457285Flushing 44e821dce55cb29b05989bca574cff62/cf: creating writer at 1676325457288 (+3 ms)Flushing mob file 44e821dce55cb29b05989bca574cff62/cf: appending metadata at 1676325457321 (+33 ms)Flushing mob file 44e821dce55cb29b05989bca574cff62/cf: closing flushed file at 1676325457321Flushing 44e821dce55cb29b05989bca574cff62/cf: appending metadata at 1676325457801 (+480 ms)Flushing 44e821dce55cb29b05989bca574cff62/cf: closing flushed file at 1676325457806 (+5 ms)Flushing 44e821dce55cb29b05989bca574cff62/cf: reopening flushed file at 1676325457875 (+69 ms)Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for 44e821dce55cb29b05989bca574cff62 in 621ms, sequenceid=8, compaction requested=false at 1676325457898 (+23 ms)Writing region close event to WAL at 1676325457929 (+31 ms)Running coprocessor post-close hooks at 1676325457937 (+8 ms)Closed at 1676325457937 2023-02-13 21:57:37,937 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 992f98999ce7b2c13e6adb6eca28ff70: Waiting for close lock at 1676325457276Running coprocessor pre-close hooks at 1676325457276Disabling compacts and flushes for region at 1676325457276Disabling writes for close at 1676325457277 (+1 ms)Obtaining lock to block concurrent updates at 1676325457277Preparing flush snapshotting stores in 992f98999ce7b2c13e6adb6eca28ff70 at 1676325457277Finished memstore snapshotting testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70., syncing WAL and waiting on mvcc, flushsize=dataSize=1807, getHeapSize=4128, getOffHeapSize=0, getCellsCount=27 at 1676325457285 (+8 ms)Flushing stores of testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. at 1676325457285Flushing 992f98999ce7b2c13e6adb6eca28ff70/cf: creating writer at 1676325457288 (+3 ms)Flushing mob file 992f98999ce7b2c13e6adb6eca28ff70/cf: appending metadata at 1676325457321 (+33 ms)Flushing mob file 992f98999ce7b2c13e6adb6eca28ff70/cf: closing flushed file at 1676325457321Flushing 992f98999ce7b2c13e6adb6eca28ff70/cf: appending metadata at 1676325457801 (+480 ms)Flushing 992f98999ce7b2c13e6adb6eca28ff70/cf: closing flushed file at 1676325457806 (+5 ms)Flushing 992f98999ce7b2c13e6adb6eca28ff70/cf: reopening flushed file at 1676325457877 (+71 ms)Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for 992f98999ce7b2c13e6adb6eca28ff70 in 621ms, sequenceid=8, compaction requested=false at 1676325457898 (+21 ms)Writing region close event to WAL at 1676325457929 (+31 ms)Running coprocessor post-close hooks at 1676325457937 (+8 ms)Closed at 1676325457937 2023-02-13 21:57:37,940 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:37,940 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:37,940 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=44e821dce55cb29b05989bca574cff62, regionState=CLOSED 2023-02-13 21:57:37,941 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 4e757d660af6a5aaa6e2c43f8b91311d, disabling compactions & flushes 2023-02-13 21:57:37,942 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:37,942 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325457940"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325457940"}]},"ts":"1676325457940"} 2023-02-13 21:57:37,942 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:37,942 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. after waiting 0 ms 2023-02-13 21:57:37,942 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:37,942 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing 4e757d660af6a5aaa6e2c43f8b91311d 1/1 column families, dataSize=2.35 KB heapSize=5.31 KB 2023-02-13 21:57:37,943 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:37,943 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:37,944 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 251bc68b009792a14e42934b238a6375, disabling compactions & flushes 2023-02-13 21:57:37,944 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:37,944 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:37,944 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. after waiting 0 ms 2023-02-13 21:57:37,944 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:37,944 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing 251bc68b009792a14e42934b238a6375 1/1 column families, dataSize=21.98 KB heapSize=47.50 KB 2023-02-13 21:57:37,944 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=992f98999ce7b2c13e6adb6eca28ff70, regionState=CLOSED 2023-02-13 21:57:37,945 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325457944"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325457944"}]},"ts":"1676325457944"} 2023-02-13 21:57:37,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/.tmp/cf/3d85d09beeee48a5b67b39f0abb4165e as hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/3d85d09beeee48a5b67b39f0abb4165e 2023-02-13 21:57:37,951 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=57, resume processing ppid=52 2023-02-13 21:57:37,951 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=57, ppid=52, state=SUCCESS; CloseRegionProcedure 44e821dce55cb29b05989bca574cff62, server=jenkins-hbase20.apache.org,45015,1676325451448 in 825 msec 2023-02-13 21:57:37,952 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=51 2023-02-13 21:57:37,953 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=51, state=SUCCESS; CloseRegionProcedure 992f98999ce7b2c13e6adb6eca28ff70, server=jenkins-hbase20.apache.org,36339,1676325451407 in 830 msec 2023-02-13 21:57:37,958 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, UNASSIGN in 841 msec 2023-02-13 21:57:37,959 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, UNASSIGN in 842 msec 2023-02-13 21:57:37,965 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/3d85d09beeee48a5b67b39f0abb4165e, entries=32, sequenceid=8, filesize=10.4 K 2023-02-13 21:57:37,981 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for 1f6dc79bf11da8444a0870d779d2fb4d in 704ms, sequenceid=8, compaction requested=false 2023-02-13 21:57:38,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 21:57:38,031 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:38,032 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 1f6dc79bf11da8444a0870d779d2fb4d: Waiting for close lock at 1676325457277Running coprocessor pre-close hooks at 1676325457277Disabling compacts and flushes for region at 1676325457277Disabling writes for close at 1676325457277Obtaining lock to block concurrent updates at 1676325457278 (+1 ms)Preparing flush snapshotting stores in 1f6dc79bf11da8444a0870d779d2fb4d at 1676325457278Finished memstore snapshotting testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d., syncing WAL and waiting on mvcc, flushsize=dataSize=2142, getHeapSize=4848, getOffHeapSize=0, getCellsCount=32 at 1676325457285 (+7 ms)Flushing stores of testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. at 1676325457285Flushing 1f6dc79bf11da8444a0870d779d2fb4d/cf: creating writer at 1676325457288 (+3 ms)Flushing mob file 1f6dc79bf11da8444a0870d779d2fb4d/cf: appending metadata at 1676325457321 (+33 ms)Flushing mob file 1f6dc79bf11da8444a0870d779d2fb4d/cf: closing flushed file at 1676325457321Flushing 1f6dc79bf11da8444a0870d779d2fb4d/cf: appending metadata at 1676325457801 (+480 ms)Flushing 1f6dc79bf11da8444a0870d779d2fb4d/cf: closing flushed file at 1676325457807 (+6 ms)Flushing 1f6dc79bf11da8444a0870d779d2fb4d/cf: reopening flushed file at 1676325457947 (+140 ms)Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for 1f6dc79bf11da8444a0870d779d2fb4d in 704ms, sequenceid=8, compaction requested=false at 1676325457981 (+34 ms)Writing region close event to WAL at 1676325458025 (+44 ms)Running coprocessor post-close hooks at 1676325458031 (+6 ms)Closed at 1676325458031 2023-02-13 21:57:38,043 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a 2023-02-13 21:57:38,044 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:38,044 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:38,044 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 676fb98d6cc62f0fdbda4ce59aad85cb, disabling compactions & flushes 2023-02-13 21:57:38,044 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:38,044 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:38,044 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. after waiting 0 ms 2023-02-13 21:57:38,044 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:38,045 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing 676fb98d6cc62f0fdbda4ce59aad85cb 1/1 column families, dataSize=2.42 KB heapSize=5.45 KB 2023-02-13 21:57:38,046 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=1f6dc79bf11da8444a0870d779d2fb4d, regionState=CLOSED 2023-02-13 21:57:38,046 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325458046"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325458046"}]},"ts":"1676325458046"} 2023-02-13 21:57:38,064 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=50 2023-02-13 21:57:38,065 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=50, state=SUCCESS; CloseRegionProcedure 1f6dc79bf11da8444a0870d779d2fb4d, server=jenkins-hbase20.apache.org,39523,1676325451480 in 936 msec 2023-02-13 21:57:38,067 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=50, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, UNASSIGN in 954 msec 2023-02-13 21:57:38,085 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e 2023-02-13 21:57:38,092 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=22.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/.tmp/cf/29f391c114814c4b96610425ae0e5de0 2023-02-13 21:57:38,100 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/.tmp/cf/29f391c114814c4b96610425ae0e5de0 as hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/29f391c114814c4b96610425ae0e5de0 2023-02-13 21:57:38,108 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/29f391c114814c4b96610425ae0e5de0, entries=336, sequenceid=8, filesize=62.2 K 2023-02-13 21:57:38,113 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~21.98 KB/22508, heapSize ~47.48 KB/48624, currentSize=0 B/0 for 251bc68b009792a14e42934b238a6375 in 168ms, sequenceid=8, compaction requested=false 2023-02-13 21:57:38,131 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b 2023-02-13 21:57:38,133 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/.tmp/cf/81bb121c83f84b0997b6675992ad85a1 2023-02-13 21:57:38,133 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 21:57:38,135 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:38,135 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 251bc68b009792a14e42934b238a6375: Waiting for close lock at 1676325457943Running coprocessor pre-close hooks at 1676325457943Disabling compacts and flushes for region at 1676325457943Disabling writes for close at 1676325457944 (+1 ms)Obtaining lock to block concurrent updates at 1676325457944Preparing flush snapshotting stores in 251bc68b009792a14e42934b238a6375 at 1676325457944Finished memstore snapshotting testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375., syncing WAL and waiting on mvcc, flushsize=dataSize=22508, getHeapSize=48624, getOffHeapSize=0, getCellsCount=336 at 1676325457944Flushing stores of testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. at 1676325457946 (+2 ms)Flushing 251bc68b009792a14e42934b238a6375/cf: creating writer at 1676325457947 (+1 ms)Flushing mob file 251bc68b009792a14e42934b238a6375/cf: appending metadata at 1676325457967 (+20 ms)Flushing mob file 251bc68b009792a14e42934b238a6375/cf: closing flushed file at 1676325457967Flushing 251bc68b009792a14e42934b238a6375/cf: appending metadata at 1676325458045 (+78 ms)Flushing 251bc68b009792a14e42934b238a6375/cf: closing flushed file at 1676325458045Flushing 251bc68b009792a14e42934b238a6375/cf: reopening flushed file at 1676325458101 (+56 ms)Finished flush of dataSize ~21.98 KB/22508, heapSize ~47.48 KB/48624, currentSize=0 B/0 for 251bc68b009792a14e42934b238a6375 in 168ms, sequenceid=8, compaction requested=false at 1676325458113 (+12 ms)Writing region close event to WAL at 1676325458122 (+9 ms)Running coprocessor post-close hooks at 1676325458135 (+13 ms)Closed at 1676325458135 2023-02-13 21:57:38,139 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=251bc68b009792a14e42934b238a6375, regionState=CLOSED 2023-02-13 21:57:38,139 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:38,139 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325458139"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325458139"}]},"ts":"1676325458139"} 2023-02-13 21:57:38,156 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/.tmp/cf/81bb121c83f84b0997b6675992ad85a1 as hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/81bb121c83f84b0997b6675992ad85a1 2023-02-13 21:57:38,158 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=55 2023-02-13 21:57:38,158 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=55, state=SUCCESS; CloseRegionProcedure 251bc68b009792a14e42934b238a6375, server=jenkins-hbase20.apache.org,36339,1676325451407 in 1.0320 sec 2023-02-13 21:57:38,161 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, UNASSIGN in 1.0490 sec 2023-02-13 21:57:38,165 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/81bb121c83f84b0997b6675992ad85a1, entries=36, sequenceid=8, filesize=11.0 K 2023-02-13 21:57:38,166 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 4e757d660af6a5aaa6e2c43f8b91311d in 224ms, sequenceid=8, compaction requested=false 2023-02-13 21:57:38,186 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/.tmp/cf/99375a84534848d588fa45ac3699c454 2023-02-13 21:57:38,187 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 21:57:38,191 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:38,191 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 4e757d660af6a5aaa6e2c43f8b91311d: Waiting for close lock at 1676325457940Running coprocessor pre-close hooks at 1676325457940Disabling compacts and flushes for region at 1676325457940Disabling writes for close at 1676325457942 (+2 ms)Obtaining lock to block concurrent updates at 1676325457942Preparing flush snapshotting stores in 4e757d660af6a5aaa6e2c43f8b91311d at 1676325457942Finished memstore snapshotting testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d., syncing WAL and waiting on mvcc, flushsize=dataSize=2410, getHeapSize=5424, getOffHeapSize=0, getCellsCount=36 at 1676325457942Flushing stores of testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. at 1676325457944 (+2 ms)Flushing 4e757d660af6a5aaa6e2c43f8b91311d/cf: creating writer at 1676325457945 (+1 ms)Flushing mob file 4e757d660af6a5aaa6e2c43f8b91311d/cf: appending metadata at 1676325457992 (+47 ms)Flushing mob file 4e757d660af6a5aaa6e2c43f8b91311d/cf: closing flushed file at 1676325457992Flushing 4e757d660af6a5aaa6e2c43f8b91311d/cf: appending metadata at 1676325458086 (+94 ms)Flushing 4e757d660af6a5aaa6e2c43f8b91311d/cf: closing flushed file at 1676325458086Flushing 4e757d660af6a5aaa6e2c43f8b91311d/cf: reopening flushed file at 1676325458157 (+71 ms)Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 4e757d660af6a5aaa6e2c43f8b91311d in 224ms, sequenceid=8, compaction requested=false at 1676325458166 (+9 ms)Writing region close event to WAL at 1676325458183 (+17 ms)Running coprocessor post-close hooks at 1676325458191 (+8 ms)Closed at 1676325458191 2023-02-13 21:57:38,193 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:38,194 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=4e757d660af6a5aaa6e2c43f8b91311d, regionState=CLOSED 2023-02-13 21:57:38,194 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325458194"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325458194"}]},"ts":"1676325458194"} 2023-02-13 21:57:38,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/.tmp/cf/99375a84534848d588fa45ac3699c454 as hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/99375a84534848d588fa45ac3699c454 2023-02-13 21:57:38,199 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=54 2023-02-13 21:57:38,199 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=54, state=SUCCESS; CloseRegionProcedure 4e757d660af6a5aaa6e2c43f8b91311d, server=jenkins-hbase20.apache.org,45015,1676325451448 in 1.0710 sec 2023-02-13 21:57:38,201 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=54, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, UNASSIGN in 1.0890 sec 2023-02-13 21:57:38,203 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/99375a84534848d588fa45ac3699c454, entries=37, sequenceid=8, filesize=11.2 K 2023-02-13 21:57:38,205 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.42 KB/2477, heapSize ~5.44 KB/5568, currentSize=0 B/0 for 676fb98d6cc62f0fdbda4ce59aad85cb in 160ms, sequenceid=8, compaction requested=false 2023-02-13 21:57:38,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 21:57:38,217 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:38,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 676fb98d6cc62f0fdbda4ce59aad85cb: Waiting for close lock at 1676325458044Running coprocessor pre-close hooks at 1676325458044Disabling compacts and flushes for region at 1676325458044Disabling writes for close at 1676325458044Obtaining lock to block concurrent updates at 1676325458045 (+1 ms)Preparing flush snapshotting stores in 676fb98d6cc62f0fdbda4ce59aad85cb at 1676325458045Finished memstore snapshotting testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb., syncing WAL and waiting on mvcc, flushsize=dataSize=2477, getHeapSize=5568, getOffHeapSize=0, getCellsCount=37 at 1676325458045Flushing stores of testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. at 1676325458046 (+1 ms)Flushing 676fb98d6cc62f0fdbda4ce59aad85cb/cf: creating writer at 1676325458055 (+9 ms)Flushing mob file 676fb98d6cc62f0fdbda4ce59aad85cb/cf: appending metadata at 1676325458074 (+19 ms)Flushing mob file 676fb98d6cc62f0fdbda4ce59aad85cb/cf: closing flushed file at 1676325458074Flushing 676fb98d6cc62f0fdbda4ce59aad85cb/cf: appending metadata at 1676325458133 (+59 ms)Flushing 676fb98d6cc62f0fdbda4ce59aad85cb/cf: closing flushed file at 1676325458133Flushing 676fb98d6cc62f0fdbda4ce59aad85cb/cf: reopening flushed file at 1676325458195 (+62 ms)Finished flush of dataSize ~2.42 KB/2477, heapSize ~5.44 KB/5568, currentSize=0 B/0 for 676fb98d6cc62f0fdbda4ce59aad85cb in 160ms, sequenceid=8, compaction requested=false at 1676325458205 (+10 ms)Writing region close event to WAL at 1676325458213 (+8 ms)Running coprocessor post-close hooks at 1676325458217 (+4 ms)Closed at 1676325458217 2023-02-13 21:57:38,220 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:38,220 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=676fb98d6cc62f0fdbda4ce59aad85cb, regionState=CLOSED 2023-02-13 21:57:38,220 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325458220"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325458220"}]},"ts":"1676325458220"} 2023-02-13 21:57:38,224 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=53 2023-02-13 21:57:38,224 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=53, state=SUCCESS; CloseRegionProcedure 676fb98d6cc62f0fdbda4ce59aad85cb, server=jenkins-hbase20.apache.org,39523,1676325451480 in 1.0820 sec 2023-02-13 21:57:38,226 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=53, resume processing ppid=49 2023-02-13 21:57:38,226 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=53, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, UNASSIGN in 1.1140 sec 2023-02-13 21:57:38,227 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325458227"}]},"ts":"1676325458227"} 2023-02-13 21:57:38,229 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676325454277, state=DISABLED in hbase:meta 2023-02-13 21:57:38,230 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676325454277 to state=DISABLED 2023-02-13 21:57:38,233 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=49, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676325454277 in 1.1320 sec 2023-02-13 21:57:38,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-13 21:57:38,612 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676325454277, procId: 49 completed 2023-02-13 21:57:38,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1580): Client=jenkins//148.251.75.209 snapshot request for:{ ss=snaptb0-1676325454277 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 } 2023-02-13 21:57:38,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676325458613 (current time:1676325458613). 2023-02-13 21:57:38,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 21:57:38,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] zookeeper.ReadOnlyZKClient(139): Connect 0x1d313954 to 127.0.0.1:58580 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 21:57:38,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7ba0a7ea, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 21:57:38,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 21:57:38,624 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34558, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 21:57:38,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d313954 to 127.0.0.1:58580 2023-02-13 21:57:38,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 21:57:38,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 21:57:38,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1676325454277 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 } 2023-02-13 21:57:38,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676325454277, type=EXCLUSIVE 2023-02-13 21:57:38,630 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676325454277, type=EXCLUSIVE 2023-02-13 21:57:38,631 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=62, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676325454277, type=EXCLUSIVE; timeout=600000, timestamp=1676326058631 2023-02-13 21:57:38,631 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1676325454277 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676325454277 2023-02-13 21:57:38,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1676325454277 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 } 2023-02-13 21:57:38,631 DEBUG [Listener at localhost.localdomain/44075] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1676325454277 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 21:57:38,631 DEBUG [Listener at localhost.localdomain/44075] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 21:57:38,648 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676325454277 table=testRestoreSnapshot-1676325454277 type=DISABLED ttl=0 } 2023-02-13 21:57:38,650 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:38,650 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:38,650 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:38,650 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:38,650 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:38,651 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:38,651 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:38,651 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:38,651 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:38,651 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:38,651 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:38,651 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:38,651 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:38,651 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:38,653 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/81bb121c83f84b0997b6675992ad85a1] hfiles 2023-02-13 21:57:38,653 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/81bb121c83f84b0997b6675992ad85a1 2023-02-13 21:57:38,654 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/3d85d09beeee48a5b67b39f0abb4165e] hfiles 2023-02-13 21:57:38,654 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/3d85d09beeee48a5b67b39f0abb4165e 2023-02-13 21:57:38,654 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/8795edd282b341fba51b1c197d77b4f9] hfiles 2023-02-13 21:57:38,654 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/ebd1b06b1c10407b903431c38f993f0f] hfiles 2023-02-13 21:57:38,654 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/8795edd282b341fba51b1c197d77b4f9 2023-02-13 21:57:38,654 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/ebd1b06b1c10407b903431c38f993f0f 2023-02-13 21:57:38,655 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/99375a84534848d588fa45ac3699c454] hfiles 2023-02-13 21:57:38,655 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/29f391c114814c4b96610425ae0e5de0] hfiles 2023-02-13 21:57:38,655 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/99375a84534848d588fa45ac3699c454 2023-02-13 21:57:38,655 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/29f391c114814c4b96610425ae0e5de0 2023-02-13 21:57:38,660 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b] hfiles 2023-02-13 21:57:38,660 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e 2023-02-13 21:57:38,660 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 2023-02-13 21:57:38,660 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 2023-02-13 21:57:38,661 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e 2023-02-13 21:57:38,661 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a 2023-02-13 21:57:38,661 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b 2023-02-13 21:57:38,684 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 21:57:38,732 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1676325454277 table=testRestoreSnapshot-1676325454277 type=DISABLED ttl=0 } as finished. 2023-02-13 21:57:38,732 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1676325454277 2023-02-13 21:57:38,733 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/snaptb0-1676325454277 2023-02-13 21:57:38,799 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-02-13 21:57:38,800 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-02-13 21:57:38,808 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/snaptb0-1676325454277 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/snaptb0-1676325454277 2023-02-13 21:57:38,846 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1676325454277 of table testRestoreSnapshot-1676325454277 completed 2023-02-13 21:57:38,846 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/snaptb0-1676325454277 2023-02-13 21:57:38,846 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/snaptb0-1676325454277 2023-02-13 21:57:38,846 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1676325454277 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676325454277 at 1676325458631Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676325454277 table=testRestoreSnapshot-1676325454277 type=DISABLED ttl=0 } at 1676325458649 (+18 ms)Consolidate snapshot: snaptb0-1676325454277 at 1676325458732 (+83 ms)Loading Region manifests for snaptb0-1676325454277 at 1676325458732Writing data manifest for snaptb0-1676325454277 at 1676325458763 (+31 ms)Verifying snapshot: snaptb0-1676325454277 at 1676325458785 (+22 ms)Snapshot snaptb0-1676325454277 of table testRestoreSnapshot-1676325454277 completed at 1676325458846 (+61 ms) 2023-02-13 21:57:38,848 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=62, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676325454277, type=EXCLUSIVE 2023-02-13 21:57:38,849 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676325454277, type=EXCLUSIVE in 220 msec 2023-02-13 21:57:38,882 DEBUG [Listener at localhost.localdomain/44075] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 21:57:38,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676325454277 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 } is done 2023-02-13 21:57:38,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1676325454277 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 21:57:38,885 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676325454277 2023-02-13 21:57:38,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.HMaster$10(2372): Client=jenkins//148.251.75.209 enable testRestoreSnapshot-1676325454277 2023-02-13 21:57:38,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=63, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:38,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-13 21:57:38,890 INFO [PEWorker-3] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676325454277 2023-02-13 21:57:38,891 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325458890"}]},"ts":"1676325458890"} 2023-02-13 21:57:38,893 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676325454277, state=ENABLING in hbase:meta 2023-02-13 21:57:38,896 DEBUG [PEWorker-3] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 21:57:38,897 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, ASSIGN}, {pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, ASSIGN}, {pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, ASSIGN}, {pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, ASSIGN}, {pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, ASSIGN}, {pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, ASSIGN}] 2023-02-13 21:57:38,900 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, ASSIGN 2023-02-13 21:57:38,900 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, ASSIGN 2023-02-13 21:57:38,900 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, ASSIGN 2023-02-13 21:57:38,900 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, ASSIGN 2023-02-13 21:57:38,903 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:38,903 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, ASSIGN 2023-02-13 21:57:38,903 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,39523,1676325451480; forceNewPlan=false, retain=true 2023-02-13 21:57:38,903 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:38,903 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:38,903 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:38,903 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,36339,1676325451407; forceNewPlan=false, retain=true 2023-02-13 21:57:38,903 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,45015,1676325451448; forceNewPlan=false, retain=true 2023-02-13 21:57:38,903 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,36339,1676325451407; forceNewPlan=false, retain=true 2023-02-13 21:57:38,905 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, ASSIGN 2023-02-13 21:57:38,905 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:38,906 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,45015,1676325451448; forceNewPlan=false, retain=true 2023-02-13 21:57:38,907 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:38,907 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,39523,1676325451480; forceNewPlan=false, retain=true 2023-02-13 21:57:39,053 INFO [jenkins-hbase20:46557] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 21:57:39,054 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=676fb98d6cc62f0fdbda4ce59aad85cb, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:39,054 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=992f98999ce7b2c13e6adb6eca28ff70, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:39,054 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=1f6dc79bf11da8444a0870d779d2fb4d, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:39,054 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=44e821dce55cb29b05989bca574cff62, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:39,054 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=4e757d660af6a5aaa6e2c43f8b91311d, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:39,054 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325459054"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325459054"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325459054"}]},"ts":"1676325459054"} 2023-02-13 21:57:39,054 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325459054"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325459054"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325459054"}]},"ts":"1676325459054"} 2023-02-13 21:57:39,054 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325459054"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325459054"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325459054"}]},"ts":"1676325459054"} 2023-02-13 21:57:39,054 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325459054"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325459054"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325459054"}]},"ts":"1676325459054"} 2023-02-13 21:57:39,054 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325459054"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325459054"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325459054"}]},"ts":"1676325459054"} 2023-02-13 21:57:39,056 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=70, ppid=66, state=RUNNABLE; OpenRegionProcedure 44e821dce55cb29b05989bca574cff62, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:57:39,057 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=71, ppid=64, state=RUNNABLE; OpenRegionProcedure 1f6dc79bf11da8444a0870d779d2fb4d, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:57:39,058 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=72, ppid=65, state=RUNNABLE; OpenRegionProcedure 992f98999ce7b2c13e6adb6eca28ff70, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:57:39,059 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=73, ppid=67, state=RUNNABLE; OpenRegionProcedure 676fb98d6cc62f0fdbda4ce59aad85cb, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:57:39,060 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=74, ppid=68, state=RUNNABLE; OpenRegionProcedure 4e757d660af6a5aaa6e2c43f8b91311d, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:57:39,067 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=251bc68b009792a14e42934b238a6375, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:39,067 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325459067"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325459067"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325459067"}]},"ts":"1676325459067"} 2023-02-13 21:57:39,073 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=75, ppid=69, state=RUNNABLE; OpenRegionProcedure 251bc68b009792a14e42934b238a6375, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:57:39,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-13 21:57:39,216 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:39,216 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 4e757d660af6a5aaa6e2c43f8b91311d, NAME => 'testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 21:57:39,216 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:39,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1f6dc79bf11da8444a0870d779d2fb4d, NAME => 'testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 21:57:39,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:39,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:39,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:39,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:39,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:39,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:39,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:39,218 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:39,220 INFO [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:39,220 INFO [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:39,222 DEBUG [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf 2023-02-13 21:57:39,222 DEBUG [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf 2023-02-13 21:57:39,222 DEBUG [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf 2023-02-13 21:57:39,222 DEBUG [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf 2023-02-13 21:57:39,223 INFO [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1f6dc79bf11da8444a0870d779d2fb4d columnFamilyName cf 2023-02-13 21:57:39,223 INFO [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4e757d660af6a5aaa6e2c43f8b91311d columnFamilyName cf 2023-02-13 21:57:39,227 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:39,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 251bc68b009792a14e42934b238a6375, NAME => 'testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 21:57:39,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:39,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:39,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:39,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:39,229 INFO [StoreOpener-251bc68b009792a14e42934b238a6375-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:39,231 DEBUG [StoreOpener-251bc68b009792a14e42934b238a6375-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf 2023-02-13 21:57:39,231 DEBUG [StoreOpener-251bc68b009792a14e42934b238a6375-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf 2023-02-13 21:57:39,232 INFO [StoreOpener-251bc68b009792a14e42934b238a6375-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 251bc68b009792a14e42934b238a6375 columnFamilyName cf 2023-02-13 21:57:39,232 DEBUG [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/3d85d09beeee48a5b67b39f0abb4165e 2023-02-13 21:57:39,233 DEBUG [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/81bb121c83f84b0997b6675992ad85a1 2023-02-13 21:57:39,233 INFO [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] regionserver.HStore(310): Store=1f6dc79bf11da8444a0870d779d2fb4d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:39,233 INFO [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] regionserver.HStore(310): Store=4e757d660af6a5aaa6e2c43f8b91311d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:39,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:39,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:39,239 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:39,239 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:39,244 DEBUG [StoreOpener-251bc68b009792a14e42934b238a6375-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/29f391c114814c4b96610425ae0e5de0 2023-02-13 21:57:39,244 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:39,244 INFO [StoreOpener-251bc68b009792a14e42934b238a6375-1] regionserver.HStore(310): Store=251bc68b009792a14e42934b238a6375/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:39,245 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:39,246 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 1f6dc79bf11da8444a0870d779d2fb4d; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74719177, jitterRate=0.11340250074863434}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:39,246 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 1f6dc79bf11da8444a0870d779d2fb4d: Running coprocessor pre-open hook at 1676325459218Writing region info on filesystem at 1676325459218Initializing all the Stores at 1676325459219 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325459219Cleaning up temporary data from old regions at 1676325459240 (+21 ms)Cleaning up detritus from prior splits at 1676325459240Running coprocessor post-open hooks at 1676325459246 (+6 ms)Region opened successfully at 1676325459246 2023-02-13 21:57:39,246 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375 2023-02-13 21:57:39,247 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d., pid=71, masterSystemTime=1676325459210 2023-02-13 21:57:39,247 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 4e757d660af6a5aaa6e2c43f8b91311d; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64624818, jitterRate=-0.037015169858932495}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:39,247 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375 2023-02-13 21:57:39,247 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 4e757d660af6a5aaa6e2c43f8b91311d: Running coprocessor pre-open hook at 1676325459218Writing region info on filesystem at 1676325459218Initializing all the Stores at 1676325459219 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325459219Cleaning up temporary data from old regions at 1676325459240 (+21 ms)Cleaning up detritus from prior splits at 1676325459241 (+1 ms)Running coprocessor post-open hooks at 1676325459247 (+6 ms)Region opened successfully at 1676325459247 2023-02-13 21:57:39,248 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d., pid=74, masterSystemTime=1676325459208 2023-02-13 21:57:39,249 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:39,249 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:39,249 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:39,249 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 676fb98d6cc62f0fdbda4ce59aad85cb, NAME => 'testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 21:57:39,250 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:39,250 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:39,250 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=1f6dc79bf11da8444a0870d779d2fb4d, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:39,250 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:39,250 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:39,250 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325459250"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325459250"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325459250"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325459250"}]},"ts":"1676325459250"} 2023-02-13 21:57:39,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:39,251 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:39,251 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:39,251 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=4e757d660af6a5aaa6e2c43f8b91311d, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:39,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 44e821dce55cb29b05989bca574cff62, NAME => 'testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 21:57:39,251 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325459251"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325459251"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325459251"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325459251"}]},"ts":"1676325459251"} 2023-02-13 21:57:39,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:39,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:39,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:39,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:39,252 INFO [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:39,254 INFO [StoreOpener-44e821dce55cb29b05989bca574cff62-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:39,254 DEBUG [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf 2023-02-13 21:57:39,254 DEBUG [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf 2023-02-13 21:57:39,255 INFO [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 676fb98d6cc62f0fdbda4ce59aad85cb columnFamilyName cf 2023-02-13 21:57:39,255 DEBUG [StoreOpener-44e821dce55cb29b05989bca574cff62-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf 2023-02-13 21:57:39,255 DEBUG [StoreOpener-44e821dce55cb29b05989bca574cff62-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf 2023-02-13 21:57:39,255 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:39,255 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=71, resume processing ppid=64 2023-02-13 21:57:39,256 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=71, ppid=64, state=SUCCESS; OpenRegionProcedure 1f6dc79bf11da8444a0870d779d2fb4d, server=jenkins-hbase20.apache.org,39523,1676325451480 in 195 msec 2023-02-13 21:57:39,256 INFO [StoreOpener-44e821dce55cb29b05989bca574cff62-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 44e821dce55cb29b05989bca574cff62 columnFamilyName cf 2023-02-13 21:57:39,257 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=74, resume processing ppid=68 2023-02-13 21:57:39,257 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=74, ppid=68, state=SUCCESS; OpenRegionProcedure 4e757d660af6a5aaa6e2c43f8b91311d, server=jenkins-hbase20.apache.org,45015,1676325451448 in 194 msec 2023-02-13 21:57:39,258 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 251bc68b009792a14e42934b238a6375; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62067463, jitterRate=-0.07512272894382477}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:39,258 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, ASSIGN in 358 msec 2023-02-13 21:57:39,258 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 251bc68b009792a14e42934b238a6375: Running coprocessor pre-open hook at 1676325459227Writing region info on filesystem at 1676325459227Initializing all the Stores at 1676325459228 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325459228Cleaning up temporary data from old regions at 1676325459248 (+20 ms)Cleaning up detritus from prior splits at 1676325459250 (+2 ms)Running coprocessor post-open hooks at 1676325459258 (+8 ms)Region opened successfully at 1676325459258 2023-02-13 21:57:39,258 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375., pid=75, masterSystemTime=1676325459216 2023-02-13 21:57:39,260 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=68, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, ASSIGN in 360 msec 2023-02-13 21:57:39,260 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:39,260 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:39,260 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:39,261 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 992f98999ce7b2c13e6adb6eca28ff70, NAME => 'testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 21:57:39,261 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=251bc68b009792a14e42934b238a6375, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:39,261 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:39,261 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325459260"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325459260"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325459260"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325459260"}]},"ts":"1676325459260"} 2023-02-13 21:57:39,261 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:39,261 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:39,261 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:39,262 INFO [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:39,264 DEBUG [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf 2023-02-13 21:57:39,264 DEBUG [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf 2023-02-13 21:57:39,264 INFO [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 992f98999ce7b2c13e6adb6eca28ff70 columnFamilyName cf 2023-02-13 21:57:39,265 DEBUG [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/99375a84534848d588fa45ac3699c454 2023-02-13 21:57:39,266 INFO [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] regionserver.HStore(310): Store=676fb98d6cc62f0fdbda4ce59aad85cb/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:39,266 DEBUG [StoreOpener-44e821dce55cb29b05989bca574cff62-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/8795edd282b341fba51b1c197d77b4f9 2023-02-13 21:57:39,266 INFO [StoreOpener-44e821dce55cb29b05989bca574cff62-1] regionserver.HStore(310): Store=44e821dce55cb29b05989bca574cff62/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:39,267 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:39,267 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:39,268 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:39,268 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:39,269 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=75, resume processing ppid=69 2023-02-13 21:57:39,269 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=75, ppid=69, state=SUCCESS; OpenRegionProcedure 251bc68b009792a14e42934b238a6375, server=jenkins-hbase20.apache.org,36339,1676325451407 in 193 msec 2023-02-13 21:57:39,271 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=69, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, ASSIGN in 372 msec 2023-02-13 21:57:39,272 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:39,272 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:39,273 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 676fb98d6cc62f0fdbda4ce59aad85cb; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60914240, jitterRate=-0.09230709075927734}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:39,273 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 44e821dce55cb29b05989bca574cff62; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67130185, jitterRate=3.177076578140259E-4}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:39,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 676fb98d6cc62f0fdbda4ce59aad85cb: Running coprocessor pre-open hook at 1676325459250Writing region info on filesystem at 1676325459250Initializing all the Stores at 1676325459252 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325459252Cleaning up temporary data from old regions at 1676325459269 (+17 ms)Cleaning up detritus from prior splits at 1676325459269Running coprocessor post-open hooks at 1676325459273 (+4 ms)Region opened successfully at 1676325459273 2023-02-13 21:57:39,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 44e821dce55cb29b05989bca574cff62: Running coprocessor pre-open hook at 1676325459252Writing region info on filesystem at 1676325459252Initializing all the Stores at 1676325459253 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325459254 (+1 ms)Cleaning up temporary data from old regions at 1676325459269 (+15 ms)Cleaning up detritus from prior splits at 1676325459269Running coprocessor post-open hooks at 1676325459273 (+4 ms)Region opened successfully at 1676325459273 2023-02-13 21:57:39,273 DEBUG [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/ebd1b06b1c10407b903431c38f993f0f 2023-02-13 21:57:39,274 INFO [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] regionserver.HStore(310): Store=992f98999ce7b2c13e6adb6eca28ff70/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:39,274 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb., pid=73, masterSystemTime=1676325459210 2023-02-13 21:57:39,274 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62., pid=70, masterSystemTime=1676325459208 2023-02-13 21:57:39,275 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:39,276 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:39,276 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:39,277 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:39,278 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=676fb98d6cc62f0fdbda4ce59aad85cb, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:39,278 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325459277"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325459277"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325459277"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325459277"}]},"ts":"1676325459277"} 2023-02-13 21:57:39,278 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:39,278 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:39,280 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=44e821dce55cb29b05989bca574cff62, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:39,280 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325459279"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325459279"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325459279"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325459279"}]},"ts":"1676325459279"} 2023-02-13 21:57:39,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:39,282 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 992f98999ce7b2c13e6adb6eca28ff70; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74976201, jitterRate=0.11723245680332184}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:39,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 992f98999ce7b2c13e6adb6eca28ff70: Running coprocessor pre-open hook at 1676325459261Writing region info on filesystem at 1676325459261Initializing all the Stores at 1676325459262 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325459262Cleaning up temporary data from old regions at 1676325459278 (+16 ms)Cleaning up detritus from prior splits at 1676325459278Running coprocessor post-open hooks at 1676325459282 (+4 ms)Region opened successfully at 1676325459282 2023-02-13 21:57:39,283 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70., pid=72, masterSystemTime=1676325459216 2023-02-13 21:57:39,284 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=73, resume processing ppid=67 2023-02-13 21:57:39,284 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=73, ppid=67, state=SUCCESS; OpenRegionProcedure 676fb98d6cc62f0fdbda4ce59aad85cb, server=jenkins-hbase20.apache.org,39523,1676325451480 in 222 msec 2023-02-13 21:57:39,285 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:39,285 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:39,285 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=70, resume processing ppid=66 2023-02-13 21:57:39,285 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=70, ppid=66, state=SUCCESS; OpenRegionProcedure 44e821dce55cb29b05989bca574cff62, server=jenkins-hbase20.apache.org,45015,1676325451448 in 226 msec 2023-02-13 21:57:39,285 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=992f98999ce7b2c13e6adb6eca28ff70, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:39,285 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325459285"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325459285"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325459285"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325459285"}]},"ts":"1676325459285"} 2023-02-13 21:57:39,287 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=67, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, ASSIGN in 387 msec 2023-02-13 21:57:39,287 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=66, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, ASSIGN in 388 msec 2023-02-13 21:57:39,289 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=72, resume processing ppid=65 2023-02-13 21:57:39,289 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=72, ppid=65, state=SUCCESS; OpenRegionProcedure 992f98999ce7b2c13e6adb6eca28ff70, server=jenkins-hbase20.apache.org,36339,1676325451407 in 229 msec 2023-02-13 21:57:39,290 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=65, resume processing ppid=63 2023-02-13 21:57:39,291 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=65, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, ASSIGN in 392 msec 2023-02-13 21:57:39,292 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325459291"}]},"ts":"1676325459291"} 2023-02-13 21:57:39,293 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676325454277, state=ENABLED in hbase:meta 2023-02-13 21:57:39,295 INFO [PEWorker-2] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676325454277' was successfully enabled. 2023-02-13 21:57:39,297 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=63, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676325454277 in 411 msec 2023-02-13 21:57:39,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-13 21:57:39,643 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676325454277, procId: 63 completed 2023-02-13 21:57:39,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 21:57:39,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 21:57:39,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 21:57:39,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 21:57:39,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 21:57:39,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 21:57:39,668 DEBUG [Listener at localhost.localdomain/44075] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676325454277 2023-02-13 21:57:39,668 DEBUG [Listener at localhost.localdomain/44075] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:39,669 INFO [Listener at localhost.localdomain/44075] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 21:57:39,953 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 21:57:40,104 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 21:57:40,311 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 21:57:40,497 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 21:57:40,512 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-13 21:57:40,513 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-02-13 21:57:40,696 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 21:57:42,435 DEBUG [Listener at localhost.localdomain/44075] zookeeper.ReadOnlyZKClient(139): Connect 0x36a3b58e to 127.0.0.1:58580 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 21:57:42,450 DEBUG [Listener at localhost.localdomain/44075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13fd136e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 21:57:42,453 DEBUG [hconnection-0x6ef78f31-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 21:57:42,457 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34574, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 21:57:42,464 DEBUG [hconnection-0x6ef78f31-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 21:57:42,467 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55828, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 21:57:42,646 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 21:57:42,815 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 21:57:42,826 DEBUG [hconnection-0x6ef78f31-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 21:57:42,834 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44976, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 21:57:42,972 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 21:57:43,168 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 21:57:43,334 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 21:57:43,640 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 21:57:44,693 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676325454277 2023-02-13 21:57:44,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.HMaster$11(2405): Client=jenkins//148.251.75.209 disable testRestoreSnapshot-1676325454277 2023-02-13 21:57:44,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=76, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:44,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-13 21:57:44,704 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325464704"}]},"ts":"1676325464704"} 2023-02-13 21:57:44,706 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676325454277, state=DISABLING in hbase:meta 2023-02-13 21:57:44,707 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676325454277 to state=DISABLING 2023-02-13 21:57:44,709 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, UNASSIGN}, {pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, UNASSIGN}, {pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, UNASSIGN}, {pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, UNASSIGN}, {pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, UNASSIGN}, {pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, UNASSIGN}] 2023-02-13 21:57:44,711 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, UNASSIGN 2023-02-13 21:57:44,713 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, UNASSIGN 2023-02-13 21:57:44,713 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, UNASSIGN 2023-02-13 21:57:44,713 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, UNASSIGN 2023-02-13 21:57:44,713 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, UNASSIGN 2023-02-13 21:57:44,714 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=251bc68b009792a14e42934b238a6375, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:44,714 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=4e757d660af6a5aaa6e2c43f8b91311d, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:44,715 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325464714"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325464714"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325464714"}]},"ts":"1676325464714"} 2023-02-13 21:57:44,715 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325464714"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325464714"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325464714"}]},"ts":"1676325464714"} 2023-02-13 21:57:44,715 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=44e821dce55cb29b05989bca574cff62, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:44,715 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325464715"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325464715"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325464715"}]},"ts":"1676325464715"} 2023-02-13 21:57:44,715 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=676fb98d6cc62f0fdbda4ce59aad85cb, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:44,715 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=992f98999ce7b2c13e6adb6eca28ff70, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:44,715 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325464715"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325464715"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325464715"}]},"ts":"1676325464715"} 2023-02-13 21:57:44,716 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325464715"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325464715"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325464715"}]},"ts":"1676325464715"} 2023-02-13 21:57:44,718 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=83, ppid=82, state=RUNNABLE; CloseRegionProcedure 251bc68b009792a14e42934b238a6375, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:57:44,720 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=84, ppid=81, state=RUNNABLE; CloseRegionProcedure 4e757d660af6a5aaa6e2c43f8b91311d, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:57:44,721 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=85, ppid=79, state=RUNNABLE; CloseRegionProcedure 44e821dce55cb29b05989bca574cff62, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:57:44,723 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=86, ppid=80, state=RUNNABLE; CloseRegionProcedure 676fb98d6cc62f0fdbda4ce59aad85cb, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:57:44,723 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, UNASSIGN 2023-02-13 21:57:44,724 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=87, ppid=78, state=RUNNABLE; CloseRegionProcedure 992f98999ce7b2c13e6adb6eca28ff70, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:57:44,724 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=1f6dc79bf11da8444a0870d779d2fb4d, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:44,724 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325464724"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325464724"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325464724"}]},"ts":"1676325464724"} 2023-02-13 21:57:44,727 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=88, ppid=77, state=RUNNABLE; CloseRegionProcedure 1f6dc79bf11da8444a0870d779d2fb4d, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:57:44,872 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:44,873 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 992f98999ce7b2c13e6adb6eca28ff70, disabling compactions & flushes 2023-02-13 21:57:44,873 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:44,873 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:44,873 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. after waiting 0 ms 2023-02-13 21:57:44,873 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:44,874 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing 992f98999ce7b2c13e6adb6eca28ff70 1/1 column families, dataSize=2.22 KB heapSize=5.03 KB 2023-02-13 21:57:44,874 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:44,875 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 44e821dce55cb29b05989bca574cff62, disabling compactions & flushes 2023-02-13 21:57:44,875 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:44,875 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:44,875 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. after waiting 0 ms 2023-02-13 21:57:44,875 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:44,875 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing 44e821dce55cb29b05989bca574cff62 1/1 column families, dataSize=1.96 KB heapSize=4.47 KB 2023-02-13 21:57:44,875 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:44,875 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 676fb98d6cc62f0fdbda4ce59aad85cb, disabling compactions & flushes 2023-02-13 21:57:44,875 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:44,875 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:44,875 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. after waiting 0 ms 2023-02-13 21:57:44,876 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:44,877 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing 676fb98d6cc62f0fdbda4ce59aad85cb 1/1 column families, dataSize=2.09 KB heapSize=4.75 KB 2023-02-13 21:57:44,925 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 2023-02-13 21:57:44,925 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 2023-02-13 21:57:44,925 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 2023-02-13 21:57:44,938 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/.tmp/cf/24e696c302c64bd2bfbbbddca37e4305 2023-02-13 21:57:44,943 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/.tmp/cf/24e696c302c64bd2bfbbbddca37e4305 as hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/24e696c302c64bd2bfbbbddca37e4305 2023-02-13 21:57:44,950 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/24e696c302c64bd2bfbbbddca37e4305, entries=32, sequenceid=15, filesize=10.4 K 2023-02-13 21:57:44,951 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for 676fb98d6cc62f0fdbda4ce59aad85cb in 74ms, sequenceid=15, compaction requested=false 2023-02-13 21:57:44,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-13 21:57:44,972 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 21:57:44,973 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:44,973 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 676fb98d6cc62f0fdbda4ce59aad85cb: Waiting for close lock at 1676325464875Running coprocessor pre-close hooks at 1676325464875Disabling compacts and flushes for region at 1676325464875Disabling writes for close at 1676325464876 (+1 ms)Obtaining lock to block concurrent updates at 1676325464877 (+1 ms)Preparing flush snapshotting stores in 676fb98d6cc62f0fdbda4ce59aad85cb at 1676325464877Finished memstore snapshotting testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb., syncing WAL and waiting on mvcc, flushsize=dataSize=2142, getHeapSize=4848, getOffHeapSize=0, getCellsCount=32 at 1676325464877Flushing stores of testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. at 1676325464883 (+6 ms)Flushing 676fb98d6cc62f0fdbda4ce59aad85cb/cf: creating writer at 1676325464903 (+20 ms)Flushing mob file 676fb98d6cc62f0fdbda4ce59aad85cb/cf: appending metadata at 1676325464911 (+8 ms)Flushing mob file 676fb98d6cc62f0fdbda4ce59aad85cb/cf: closing flushed file at 1676325464911Flushing 676fb98d6cc62f0fdbda4ce59aad85cb/cf: appending metadata at 1676325464927 (+16 ms)Flushing 676fb98d6cc62f0fdbda4ce59aad85cb/cf: closing flushed file at 1676325464927Flushing 676fb98d6cc62f0fdbda4ce59aad85cb/cf: reopening flushed file at 1676325464944 (+17 ms)Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for 676fb98d6cc62f0fdbda4ce59aad85cb in 74ms, sequenceid=15, compaction requested=false at 1676325464951 (+7 ms)Writing region close event to WAL at 1676325464968 (+17 ms)Running coprocessor post-close hooks at 1676325464972 (+4 ms)Closed at 1676325464973 (+1 ms) 2023-02-13 21:57:44,975 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:44,975 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:44,975 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 1f6dc79bf11da8444a0870d779d2fb4d, disabling compactions & flushes 2023-02-13 21:57:44,976 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:44,976 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=676fb98d6cc62f0fdbda4ce59aad85cb, regionState=CLOSED 2023-02-13 21:57:44,976 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:44,976 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. after waiting 0 ms 2023-02-13 21:57:44,976 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:44,976 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325464976"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325464976"}]},"ts":"1676325464976"} 2023-02-13 21:57:44,976 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing 1f6dc79bf11da8444a0870d779d2fb4d 1/1 column families, dataSize=1.90 KB heapSize=4.33 KB 2023-02-13 21:57:44,979 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=86, resume processing ppid=80 2023-02-13 21:57:44,979 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=86, ppid=80, state=SUCCESS; CloseRegionProcedure 676fb98d6cc62f0fdbda4ce59aad85cb, server=jenkins-hbase20.apache.org,39523,1676325451480 in 254 msec 2023-02-13 21:57:44,980 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=80, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, UNASSIGN in 270 msec 2023-02-13 21:57:44,999 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 2023-02-13 21:57:45,015 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/.tmp/cf/ecd3475741414bf0a2161d2b27bc209a 2023-02-13 21:57:45,018 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/.tmp/cf/ecd3475741414bf0a2161d2b27bc209a as hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/ecd3475741414bf0a2161d2b27bc209a 2023-02-13 21:57:45,022 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/ecd3475741414bf0a2161d2b27bc209a, entries=29, sequenceid=15, filesize=9.8 K 2023-02-13 21:57:45,023 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for 1f6dc79bf11da8444a0870d779d2fb4d in 47ms, sequenceid=15, compaction requested=false 2023-02-13 21:57:45,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 21:57:45,031 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:45,032 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 1f6dc79bf11da8444a0870d779d2fb4d: Waiting for close lock at 1676325464975Running coprocessor pre-close hooks at 1676325464975Disabling compacts and flushes for region at 1676325464975Disabling writes for close at 1676325464976 (+1 ms)Obtaining lock to block concurrent updates at 1676325464976Preparing flush snapshotting stores in 1f6dc79bf11da8444a0870d779d2fb4d at 1676325464976Finished memstore snapshotting testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d., syncing WAL and waiting on mvcc, flushsize=dataSize=1941, getHeapSize=4416, getOffHeapSize=0, getCellsCount=29 at 1676325464976Flushing stores of testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. at 1676325464983 (+7 ms)Flushing 1f6dc79bf11da8444a0870d779d2fb4d/cf: creating writer at 1676325464984 (+1 ms)Flushing mob file 1f6dc79bf11da8444a0870d779d2fb4d/cf: appending metadata at 1676325464989 (+5 ms)Flushing mob file 1f6dc79bf11da8444a0870d779d2fb4d/cf: closing flushed file at 1676325464989Flushing 1f6dc79bf11da8444a0870d779d2fb4d/cf: appending metadata at 1676325465000 (+11 ms)Flushing 1f6dc79bf11da8444a0870d779d2fb4d/cf: closing flushed file at 1676325465000Flushing 1f6dc79bf11da8444a0870d779d2fb4d/cf: reopening flushed file at 1676325465019 (+19 ms)Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for 1f6dc79bf11da8444a0870d779d2fb4d in 47ms, sequenceid=15, compaction requested=false at 1676325465023 (+4 ms)Writing region close event to WAL at 1676325465027 (+4 ms)Running coprocessor post-close hooks at 1676325465031 (+4 ms)Closed at 1676325465031 2023-02-13 21:57:45,033 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:45,034 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=1f6dc79bf11da8444a0870d779d2fb4d, regionState=CLOSED 2023-02-13 21:57:45,034 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325465034"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325465034"}]},"ts":"1676325465034"} 2023-02-13 21:57:45,037 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=88, resume processing ppid=77 2023-02-13 21:57:45,037 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=88, ppid=77, state=SUCCESS; CloseRegionProcedure 1f6dc79bf11da8444a0870d779d2fb4d, server=jenkins-hbase20.apache.org,39523,1676325451480 in 309 msec 2023-02-13 21:57:45,038 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=77, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, UNASSIGN in 328 msec 2023-02-13 21:57:45,339 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/.tmp/cf/f9ada1ebb3d0446a9743d17910965899 2023-02-13 21:57:45,339 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/.tmp/cf/46bad1cc5702482388220a80e425cbe0 2023-02-13 21:57:45,348 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/.tmp/cf/f9ada1ebb3d0446a9743d17910965899 as hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/f9ada1ebb3d0446a9743d17910965899 2023-02-13 21:57:45,348 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/.tmp/cf/46bad1cc5702482388220a80e425cbe0 as hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/46bad1cc5702482388220a80e425cbe0 2023-02-13 21:57:45,352 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/f9ada1ebb3d0446a9743d17910965899, entries=30, sequenceid=15, filesize=10.0 K 2023-02-13 21:57:45,352 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/46bad1cc5702482388220a80e425cbe0, entries=34, sequenceid=15, filesize=10.7 K 2023-02-13 21:57:45,353 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.22 KB/2276, heapSize ~5.02 KB/5136, currentSize=0 B/0 for 992f98999ce7b2c13e6adb6eca28ff70 in 480ms, sequenceid=15, compaction requested=false 2023-02-13 21:57:45,353 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for 44e821dce55cb29b05989bca574cff62 in 478ms, sequenceid=15, compaction requested=false 2023-02-13 21:57:45,360 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 21:57:45,360 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 21:57:45,361 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:45,361 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:45,361 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 44e821dce55cb29b05989bca574cff62: Waiting for close lock at 1676325464874Running coprocessor pre-close hooks at 1676325464874Disabling compacts and flushes for region at 1676325464874Disabling writes for close at 1676325464875 (+1 ms)Obtaining lock to block concurrent updates at 1676325464875Preparing flush snapshotting stores in 44e821dce55cb29b05989bca574cff62 at 1676325464875Finished memstore snapshotting testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62., syncing WAL and waiting on mvcc, flushsize=dataSize=2008, getHeapSize=4560, getOffHeapSize=0, getCellsCount=30 at 1676325464875Flushing stores of testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. at 1676325464878 (+3 ms)Flushing 44e821dce55cb29b05989bca574cff62/cf: creating writer at 1676325464879 (+1 ms)Flushing mob file 44e821dce55cb29b05989bca574cff62/cf: appending metadata at 1676325464909 (+30 ms)Flushing mob file 44e821dce55cb29b05989bca574cff62/cf: closing flushed file at 1676325464909Flushing 44e821dce55cb29b05989bca574cff62/cf: appending metadata at 1676325464927 (+18 ms)Flushing 44e821dce55cb29b05989bca574cff62/cf: closing flushed file at 1676325464927Flushing 44e821dce55cb29b05989bca574cff62/cf: reopening flushed file at 1676325465349 (+422 ms)Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for 44e821dce55cb29b05989bca574cff62 in 478ms, sequenceid=15, compaction requested=false at 1676325465353 (+4 ms)Writing region close event to WAL at 1676325465357 (+4 ms)Running coprocessor post-close hooks at 1676325465360 (+3 ms)Closed at 1676325465361 (+1 ms) 2023-02-13 21:57:45,361 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 992f98999ce7b2c13e6adb6eca28ff70: Waiting for close lock at 1676325464872Running coprocessor pre-close hooks at 1676325464872Disabling compacts and flushes for region at 1676325464872Disabling writes for close at 1676325464873 (+1 ms)Obtaining lock to block concurrent updates at 1676325464874 (+1 ms)Preparing flush snapshotting stores in 992f98999ce7b2c13e6adb6eca28ff70 at 1676325464874Finished memstore snapshotting testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70., syncing WAL and waiting on mvcc, flushsize=dataSize=2276, getHeapSize=5136, getOffHeapSize=0, getCellsCount=34 at 1676325464875 (+1 ms)Flushing stores of testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. at 1676325464875Flushing 992f98999ce7b2c13e6adb6eca28ff70/cf: creating writer at 1676325464876 (+1 ms)Flushing mob file 992f98999ce7b2c13e6adb6eca28ff70/cf: appending metadata at 1676325464909 (+33 ms)Flushing mob file 992f98999ce7b2c13e6adb6eca28ff70/cf: closing flushed file at 1676325464909Flushing 992f98999ce7b2c13e6adb6eca28ff70/cf: appending metadata at 1676325464926 (+17 ms)Flushing 992f98999ce7b2c13e6adb6eca28ff70/cf: closing flushed file at 1676325464926Flushing 992f98999ce7b2c13e6adb6eca28ff70/cf: reopening flushed file at 1676325465349 (+423 ms)Finished flush of dataSize ~2.22 KB/2276, heapSize ~5.02 KB/5136, currentSize=0 B/0 for 992f98999ce7b2c13e6adb6eca28ff70 in 480ms, sequenceid=15, compaction requested=false at 1676325465353 (+4 ms)Writing region close event to WAL at 1676325465357 (+4 ms)Running coprocessor post-close hooks at 1676325465361 (+4 ms)Closed at 1676325465361 2023-02-13 21:57:45,363 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:45,363 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:45,364 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 4e757d660af6a5aaa6e2c43f8b91311d, disabling compactions & flushes 2023-02-13 21:57:45,364 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:45,364 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:45,364 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. after waiting 0 ms 2023-02-13 21:57:45,364 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:45,364 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=44e821dce55cb29b05989bca574cff62, regionState=CLOSED 2023-02-13 21:57:45,364 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing 4e757d660af6a5aaa6e2c43f8b91311d 1/1 column families, dataSize=2.55 KB heapSize=5.73 KB 2023-02-13 21:57:45,364 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325465364"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325465364"}]},"ts":"1676325465364"} 2023-02-13 21:57:45,364 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:45,365 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:45,365 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 251bc68b009792a14e42934b238a6375, disabling compactions & flushes 2023-02-13 21:57:45,365 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:45,365 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:45,365 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. after waiting 0 ms 2023-02-13 21:57:45,366 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:45,366 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing 251bc68b009792a14e42934b238a6375 1/1 column families, dataSize=21.98 KB heapSize=47.50 KB 2023-02-13 21:57:45,366 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=992f98999ce7b2c13e6adb6eca28ff70, regionState=CLOSED 2023-02-13 21:57:45,367 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325465366"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325465366"}]},"ts":"1676325465366"} 2023-02-13 21:57:45,371 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=85, resume processing ppid=79 2023-02-13 21:57:45,371 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=85, ppid=79, state=SUCCESS; CloseRegionProcedure 44e821dce55cb29b05989bca574cff62, server=jenkins-hbase20.apache.org,45015,1676325451448 in 647 msec 2023-02-13 21:57:45,371 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=87, resume processing ppid=78 2023-02-13 21:57:45,371 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=87, ppid=78, state=SUCCESS; CloseRegionProcedure 992f98999ce7b2c13e6adb6eca28ff70, server=jenkins-hbase20.apache.org,36339,1676325451407 in 645 msec 2023-02-13 21:57:45,372 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=79, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, UNASSIGN in 662 msec 2023-02-13 21:57:45,373 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=78, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, UNASSIGN in 662 msec 2023-02-13 21:57:45,390 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 2023-02-13 21:57:45,393 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 2023-02-13 21:57:45,399 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/.tmp/cf/08e8b394e04742929b62615c32c8ed4c 2023-02-13 21:57:45,401 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=22.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/.tmp/cf/136600ed36234d64b213b50e0d6f02a9 2023-02-13 21:57:45,403 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/.tmp/cf/08e8b394e04742929b62615c32c8ed4c as hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/08e8b394e04742929b62615c32c8ed4c 2023-02-13 21:57:45,404 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/.tmp/cf/136600ed36234d64b213b50e0d6f02a9 as hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/136600ed36234d64b213b50e0d6f02a9 2023-02-13 21:57:45,407 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/08e8b394e04742929b62615c32c8ed4c, entries=39, sequenceid=15, filesize=11.5 K 2023-02-13 21:57:45,408 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/136600ed36234d64b213b50e0d6f02a9, entries=336, sequenceid=15, filesize=62.2 K 2023-02-13 21:57:45,409 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~21.98 KB/22508, heapSize ~47.48 KB/48624, currentSize=0 B/0 for 251bc68b009792a14e42934b238a6375 in 43ms, sequenceid=15, compaction requested=false 2023-02-13 21:57:45,414 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.55 KB/2611, heapSize ~5.72 KB/5856, currentSize=0 B/0 for 4e757d660af6a5aaa6e2c43f8b91311d in 50ms, sequenceid=15, compaction requested=false 2023-02-13 21:57:45,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 21:57:45,421 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:45,421 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 21:57:45,421 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 251bc68b009792a14e42934b238a6375: Waiting for close lock at 1676325465365Running coprocessor pre-close hooks at 1676325465365Disabling compacts and flushes for region at 1676325465365Disabling writes for close at 1676325465366 (+1 ms)Obtaining lock to block concurrent updates at 1676325465366Preparing flush snapshotting stores in 251bc68b009792a14e42934b238a6375 at 1676325465366Finished memstore snapshotting testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375., syncing WAL and waiting on mvcc, flushsize=dataSize=22508, getHeapSize=48624, getOffHeapSize=0, getCellsCount=336 at 1676325465367 (+1 ms)Flushing stores of testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. at 1676325465367Flushing 251bc68b009792a14e42934b238a6375/cf: creating writer at 1676325465368 (+1 ms)Flushing mob file 251bc68b009792a14e42934b238a6375/cf: appending metadata at 1676325465376 (+8 ms)Flushing mob file 251bc68b009792a14e42934b238a6375/cf: closing flushed file at 1676325465376Flushing 251bc68b009792a14e42934b238a6375/cf: appending metadata at 1676325465394 (+18 ms)Flushing 251bc68b009792a14e42934b238a6375/cf: closing flushed file at 1676325465394Flushing 251bc68b009792a14e42934b238a6375/cf: reopening flushed file at 1676325465405 (+11 ms)Finished flush of dataSize ~21.98 KB/22508, heapSize ~47.48 KB/48624, currentSize=0 B/0 for 251bc68b009792a14e42934b238a6375 in 43ms, sequenceid=15, compaction requested=false at 1676325465409 (+4 ms)Writing region close event to WAL at 1676325465413 (+4 ms)Running coprocessor post-close hooks at 1676325465421 (+8 ms)Closed at 1676325465421 2023-02-13 21:57:45,423 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:45,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 4e757d660af6a5aaa6e2c43f8b91311d: Waiting for close lock at 1676325465363Running coprocessor pre-close hooks at 1676325465363Disabling compacts and flushes for region at 1676325465363Disabling writes for close at 1676325465364 (+1 ms)Obtaining lock to block concurrent updates at 1676325465364Preparing flush snapshotting stores in 4e757d660af6a5aaa6e2c43f8b91311d at 1676325465364Finished memstore snapshotting testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d., syncing WAL and waiting on mvcc, flushsize=dataSize=2611, getHeapSize=5856, getOffHeapSize=0, getCellsCount=39 at 1676325465365 (+1 ms)Flushing stores of testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. at 1676325465366 (+1 ms)Flushing 4e757d660af6a5aaa6e2c43f8b91311d/cf: creating writer at 1676325465366Flushing mob file 4e757d660af6a5aaa6e2c43f8b91311d/cf: appending metadata at 1676325465372 (+6 ms)Flushing mob file 4e757d660af6a5aaa6e2c43f8b91311d/cf: closing flushed file at 1676325465372Flushing 4e757d660af6a5aaa6e2c43f8b91311d/cf: appending metadata at 1676325465391 (+19 ms)Flushing 4e757d660af6a5aaa6e2c43f8b91311d/cf: closing flushed file at 1676325465391Flushing 4e757d660af6a5aaa6e2c43f8b91311d/cf: reopening flushed file at 1676325465404 (+13 ms)Finished flush of dataSize ~2.55 KB/2611, heapSize ~5.72 KB/5856, currentSize=0 B/0 for 4e757d660af6a5aaa6e2c43f8b91311d in 50ms, sequenceid=15, compaction requested=false at 1676325465414 (+10 ms)Writing region close event to WAL at 1676325465417 (+3 ms)Running coprocessor post-close hooks at 1676325465422 (+5 ms)Closed at 1676325465423 (+1 ms) 2023-02-13 21:57:45,424 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:45,425 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=251bc68b009792a14e42934b238a6375, regionState=CLOSED 2023-02-13 21:57:45,425 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:45,425 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325465424"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325465424"}]},"ts":"1676325465424"} 2023-02-13 21:57:45,426 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=4e757d660af6a5aaa6e2c43f8b91311d, regionState=CLOSED 2023-02-13 21:57:45,426 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325465425"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325465425"}]},"ts":"1676325465425"} 2023-02-13 21:57:45,429 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=83, resume processing ppid=82 2023-02-13 21:57:45,429 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=83, ppid=82, state=SUCCESS; CloseRegionProcedure 251bc68b009792a14e42934b238a6375, server=jenkins-hbase20.apache.org,36339,1676325451407 in 709 msec 2023-02-13 21:57:45,429 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=84, resume processing ppid=81 2023-02-13 21:57:45,429 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=84, ppid=81, state=SUCCESS; CloseRegionProcedure 4e757d660af6a5aaa6e2c43f8b91311d, server=jenkins-hbase20.apache.org,45015,1676325451448 in 708 msec 2023-02-13 21:57:45,430 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=82, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, UNASSIGN in 720 msec 2023-02-13 21:57:45,430 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=81, resume processing ppid=76 2023-02-13 21:57:45,431 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=81, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, UNASSIGN in 720 msec 2023-02-13 21:57:45,431 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325465431"}]},"ts":"1676325465431"} 2023-02-13 21:57:45,432 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676325454277, state=DISABLED in hbase:meta 2023-02-13 21:57:45,433 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676325454277 to state=DISABLED 2023-02-13 21:57:45,435 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=76, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676325454277 in 740 msec 2023-02-13 21:57:45,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-13 21:57:45,457 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676325454277, procId: 76 completed 2023-02-13 21:57:45,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1580): Client=jenkins//148.251.75.209 snapshot request for:{ ss=snaptb1-1676325454277 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 } 2023-02-13 21:57:45,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676325465457 (current time:1676325465457). 2023-02-13 21:57:45,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 21:57:45,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] zookeeper.ReadOnlyZKClient(139): Connect 0x715b6adb to 127.0.0.1:58580 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 21:57:45,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6534163d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 21:57:45,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 21:57:45,466 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:53030, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 21:57:45,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x715b6adb to 127.0.0.1:58580 2023-02-13 21:57:45,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 21:57:45,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 21:57:45,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb1-1676325454277 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 } 2023-02-13 21:57:45,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676325454277, type=EXCLUSIVE 2023-02-13 21:57:45,470 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676325454277, type=EXCLUSIVE 2023-02-13 21:57:45,471 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=89, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676325454277, type=EXCLUSIVE; timeout=600000, timestamp=1676326065471 2023-02-13 21:57:45,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb1-1676325454277 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 } 2023-02-13 21:57:45,471 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb1-1676325454277 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676325454277 2023-02-13 21:57:45,471 DEBUG [Listener at localhost.localdomain/44075] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb1-1676325454277 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 21:57:45,472 DEBUG [Listener at localhost.localdomain/44075] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 21:57:45,484 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1676325454277 table=testRestoreSnapshot-1676325454277 type=DISABLED ttl=0 } 2023-02-13 21:57:45,488 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:45,488 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:45,488 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:45,488 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:45,488 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:45,488 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:45,488 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:45,488 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:45,489 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:45,489 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:45,489 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:45,489 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:45,488 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:45,489 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:45,490 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/8795edd282b341fba51b1c197d77b4f9, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/f9ada1ebb3d0446a9743d17910965899] hfiles 2023-02-13 21:57:45,490 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/8795edd282b341fba51b1c197d77b4f9 2023-02-13 21:57:45,490 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/f9ada1ebb3d0446a9743d17910965899 2023-02-13 21:57:45,492 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/46bad1cc5702482388220a80e425cbe0, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/ebd1b06b1c10407b903431c38f993f0f] hfiles 2023-02-13 21:57:45,492 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/46bad1cc5702482388220a80e425cbe0 2023-02-13 21:57:45,492 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/ebd1b06b1c10407b903431c38f993f0f 2023-02-13 21:57:45,492 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/24e696c302c64bd2bfbbbddca37e4305, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/99375a84534848d588fa45ac3699c454] hfiles 2023-02-13 21:57:45,492 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/24e696c302c64bd2bfbbbddca37e4305 2023-02-13 21:57:45,492 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/99375a84534848d588fa45ac3699c454 2023-02-13 21:57:45,493 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/3d85d09beeee48a5b67b39f0abb4165e, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/ecd3475741414bf0a2161d2b27bc209a] hfiles 2023-02-13 21:57:45,493 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/3d85d09beeee48a5b67b39f0abb4165e 2023-02-13 21:57:45,493 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/ecd3475741414bf0a2161d2b27bc209a 2023-02-13 21:57:45,493 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/08e8b394e04742929b62615c32c8ed4c, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/81bb121c83f84b0997b6675992ad85a1] hfiles 2023-02-13 21:57:45,493 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/08e8b394e04742929b62615c32c8ed4c 2023-02-13 21:57:45,493 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/81bb121c83f84b0997b6675992ad85a1 2023-02-13 21:57:45,495 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/136600ed36234d64b213b50e0d6f02a9, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/29f391c114814c4b96610425ae0e5de0] hfiles 2023-02-13 21:57:45,495 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/136600ed36234d64b213b50e0d6f02a9 2023-02-13 21:57:45,495 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/29f391c114814c4b96610425ae0e5de0 2023-02-13 21:57:45,501 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b] hfiles 2023-02-13 21:57:45,502 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e 2023-02-13 21:57:45,502 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 2023-02-13 21:57:45,502 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 2023-02-13 21:57:45,502 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 2023-02-13 21:57:45,502 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 2023-02-13 21:57:45,502 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 2023-02-13 21:57:45,502 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e 2023-02-13 21:57:45,502 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 2023-02-13 21:57:45,502 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a 2023-02-13 21:57:45,502 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 2023-02-13 21:57:45,502 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 2023-02-13 21:57:45,502 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b 2023-02-13 21:57:45,535 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb1-1676325454277 table=testRestoreSnapshot-1676325454277 type=DISABLED ttl=0 } as finished. 2023-02-13 21:57:45,535 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb1-1676325454277 2023-02-13 21:57:45,536 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/snaptb1-1676325454277 2023-02-13 21:57:45,558 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/snaptb1-1676325454277 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/snaptb1-1676325454277 2023-02-13 21:57:45,575 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb1-1676325454277 of table testRestoreSnapshot-1676325454277 completed 2023-02-13 21:57:45,576 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/snaptb1-1676325454277 2023-02-13 21:57:45,577 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/snaptb1-1676325454277 2023-02-13 21:57:45,577 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb1-1676325454277 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676325454277 at 1676325465471Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1676325454277 table=testRestoreSnapshot-1676325454277 type=DISABLED ttl=0 } at 1676325465484 (+13 ms)Consolidate snapshot: snaptb1-1676325454277 at 1676325465535 (+51 ms)Loading Region manifests for snaptb1-1676325454277 at 1676325465535Writing data manifest for snaptb1-1676325454277 at 1676325465538 (+3 ms)Verifying snapshot: snaptb1-1676325454277 at 1676325465546 (+8 ms)Snapshot snaptb1-1676325454277 of table testRestoreSnapshot-1676325454277 completed at 1676325465575 (+29 ms) 2023-02-13 21:57:45,578 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=89, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676325454277, type=EXCLUSIVE 2023-02-13 21:57:45,579 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=89, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676325454277, type=EXCLUSIVE in 109 msec 2023-02-13 21:57:45,722 DEBUG [Listener at localhost.localdomain/44075] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 21:57:45,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb1-1676325454277 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 } is done 2023-02-13 21:57:45,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb1-1676325454277 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 21:57:45,744 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb0-1676325454277-1676325465744 2023-02-13 21:57:45,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1580): Client=jenkins//148.251.75.209 snapshot request for:{ ss=hbase-failsafe-snaptb0-1676325454277-1676325465744 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 } 2023-02-13 21:57:45,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676325465746 (current time:1676325465746). 2023-02-13 21:57:45,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 21:57:45,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] zookeeper.ReadOnlyZKClient(139): Connect 0x4c0c7b3e to 127.0.0.1:58580 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 21:57:45,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3a6f5353, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 21:57:45,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 21:57:45,755 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:53046, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 21:57:45,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4c0c7b3e to 127.0.0.1:58580 2023-02-13 21:57:45,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 21:57:45,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 21:57:45,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb0-1676325454277-1676325465744 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 } 2023-02-13 21:57:45,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676325454277, type=EXCLUSIVE 2023-02-13 21:57:45,759 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676325454277, type=EXCLUSIVE 2023-02-13 21:57:45,760 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=90, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676325454277, type=EXCLUSIVE; timeout=600000, timestamp=1676326065760 2023-02-13 21:57:45,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb0-1676325454277-1676325465744 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 } 2023-02-13 21:57:45,760 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb0-1676325454277-1676325465744 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676325454277 2023-02-13 21:57:45,760 DEBUG [Listener at localhost.localdomain/44075] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb0-1676325454277-1676325465744 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 21:57:45,760 DEBUG [Listener at localhost.localdomain/44075] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 21:57:45,771 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1676325454277-1676325465744 table=testRestoreSnapshot-1676325454277 type=DISABLED ttl=0 } 2023-02-13 21:57:45,773 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:45,774 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:45,774 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:45,774 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:45,774 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:45,774 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:45,774 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:45,774 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:45,774 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:45,774 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:45,774 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:45,774 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:45,774 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:45,774 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:45,776 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/08e8b394e04742929b62615c32c8ed4c, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/81bb121c83f84b0997b6675992ad85a1] hfiles 2023-02-13 21:57:45,776 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/08e8b394e04742929b62615c32c8ed4c 2023-02-13 21:57:45,776 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/81bb121c83f84b0997b6675992ad85a1 2023-02-13 21:57:45,776 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/136600ed36234d64b213b50e0d6f02a9, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/29f391c114814c4b96610425ae0e5de0] hfiles 2023-02-13 21:57:45,776 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/136600ed36234d64b213b50e0d6f02a9 2023-02-13 21:57:45,776 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/29f391c114814c4b96610425ae0e5de0 2023-02-13 21:57:45,778 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/3d85d09beeee48a5b67b39f0abb4165e, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/ecd3475741414bf0a2161d2b27bc209a] hfiles 2023-02-13 21:57:45,778 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/3d85d09beeee48a5b67b39f0abb4165e 2023-02-13 21:57:45,778 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/ecd3475741414bf0a2161d2b27bc209a 2023-02-13 21:57:45,779 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/24e696c302c64bd2bfbbbddca37e4305, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/99375a84534848d588fa45ac3699c454] hfiles 2023-02-13 21:57:45,779 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/46bad1cc5702482388220a80e425cbe0, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/ebd1b06b1c10407b903431c38f993f0f] hfiles 2023-02-13 21:57:45,779 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/8795edd282b341fba51b1c197d77b4f9, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/f9ada1ebb3d0446a9743d17910965899] hfiles 2023-02-13 21:57:45,779 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/46bad1cc5702482388220a80e425cbe0 2023-02-13 21:57:45,779 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/24e696c302c64bd2bfbbbddca37e4305 2023-02-13 21:57:45,779 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/ebd1b06b1c10407b903431c38f993f0f 2023-02-13 21:57:45,779 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/99375a84534848d588fa45ac3699c454 2023-02-13 21:57:45,779 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/8795edd282b341fba51b1c197d77b4f9 2023-02-13 21:57:45,779 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/f9ada1ebb3d0446a9743d17910965899 2023-02-13 21:57:45,789 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b] hfiles 2023-02-13 21:57:45,789 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e 2023-02-13 21:57:45,789 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 2023-02-13 21:57:45,789 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 2023-02-13 21:57:45,789 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 2023-02-13 21:57:45,789 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 2023-02-13 21:57:45,789 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 2023-02-13 21:57:45,789 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e 2023-02-13 21:57:45,789 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 2023-02-13 21:57:45,790 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a 2023-02-13 21:57:45,790 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 2023-02-13 21:57:45,790 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 2023-02-13 21:57:45,790 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b 2023-02-13 21:57:45,797 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb0-1676325454277-1676325465744 table=testRestoreSnapshot-1676325454277 type=DISABLED ttl=0 } as finished. 2023-02-13 21:57:45,798 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb0-1676325454277-1676325465744 2023-02-13 21:57:45,798 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676325454277-1676325465744 2023-02-13 21:57:45,834 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676325454277-1676325465744 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/hbase-failsafe-snaptb0-1676325454277-1676325465744 2023-02-13 21:57:45,854 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb0-1676325454277-1676325465744 of table testRestoreSnapshot-1676325454277 completed 2023-02-13 21:57:45,854 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676325454277-1676325465744 2023-02-13 21:57:45,854 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676325454277-1676325465744 2023-02-13 21:57:45,854 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb0-1676325454277-1676325465744 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676325454277 at 1676325465760Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1676325454277-1676325465744 table=testRestoreSnapshot-1676325454277 type=DISABLED ttl=0 } at 1676325465772 (+12 ms)Consolidate snapshot: hbase-failsafe-snaptb0-1676325454277-1676325465744 at 1676325465798 (+26 ms)Loading Region manifests for hbase-failsafe-snaptb0-1676325454277-1676325465744 at 1676325465798Writing data manifest for hbase-failsafe-snaptb0-1676325454277-1676325465744 at 1676325465806 (+8 ms)Verifying snapshot: hbase-failsafe-snaptb0-1676325454277-1676325465744 at 1676325465821 (+15 ms)Snapshot hbase-failsafe-snaptb0-1676325454277-1676325465744 of table testRestoreSnapshot-1676325454277 completed at 1676325465854 (+33 ms) 2023-02-13 21:57:45,856 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=90, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676325454277, type=EXCLUSIVE 2023-02-13 21:57:45,857 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=90, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676325454277, type=EXCLUSIVE in 99 msec 2023-02-13 21:57:46,010 DEBUG [Listener at localhost.localdomain/44075] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 21:57:46,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb0-1676325454277-1676325465744 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 } is done 2023-02-13 21:57:46,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb0-1676325454277-1676325465744 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 21:57:46,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=91, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676325454277 snapshot=name: "snaptb0-1676325454277" table: "testRestoreSnapshot-1676325454277" creation_time: 1676325458613 type: DISABLED version: 2 ttl: 0 ) 2023-02-13 21:57:46,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(880): Restore snapshot=snaptb0-1676325454277 as table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:46,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-02-13 21:57:46,056 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/.tabledesc/.tableinfo.0000000002 2023-02-13 21:57:46,057 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/.tabledesc/.tableinfo.0000000001 2023-02-13 21:57:46,057 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/.tabledesc/.tableinfo.0000000002 2023-02-13 21:57:46,058 INFO [PEWorker-4] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb0-1676325454277 table=testRestoreSnapshot-1676325454277 type=DISABLED ttl=0 } 2023-02-13 21:57:46,062 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb0-1676325454277" table: "testRestoreSnapshot-1676325454277" creation_time: 1676325458613 type: DISABLED version: 2 ttl: 0 2023-02-13 21:57:46,063 DEBUG [PEWorker-4] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277 2023-02-13 21:57:46,076 DEBUG [PEWorker-4] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:46,085 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 2023-02-13 21:57:46,087 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 2023-02-13 21:57:46,089 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 2023-02-13 21:57:46,090 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 2023-02-13 21:57:46,091 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 2023-02-13 21:57:46,092 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 2023-02-13 21:57:46,092 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:46,092 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:46,092 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:46,092 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:46,092 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:46,092 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:46,097 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/f9ada1ebb3d0446a9743d17910965899 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/f9ada1ebb3d0446a9743d17910965899 2023-02-13 21:57:46,097 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/24e696c302c64bd2bfbbbddca37e4305 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/24e696c302c64bd2bfbbbddca37e4305 2023-02-13 21:57:46,097 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/08e8b394e04742929b62615c32c8ed4c to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/08e8b394e04742929b62615c32c8ed4c 2023-02-13 21:57:46,097 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/46bad1cc5702482388220a80e425cbe0 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/46bad1cc5702482388220a80e425cbe0 2023-02-13 21:57:46,097 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/ecd3475741414bf0a2161d2b27bc209a to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/ecd3475741414bf0a2161d2b27bc209a 2023-02-13 21:57:46,098 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/136600ed36234d64b213b50e0d6f02a9 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/136600ed36234d64b213b50e0d6f02a9 2023-02-13 21:57:46,099 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb0-1676325454277" table: "testRestoreSnapshot-1676325454277" creation_time: 1676325458613 type: DISABLED version: 2 ttl: 0 2023-02-13 21:57:46,101 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325466100"}]},"ts":"9223372036854775807"} 2023-02-13 21:57:46,101 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325466100"}]},"ts":"9223372036854775807"} 2023-02-13 21:57:46,101 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325466100"}]},"ts":"9223372036854775807"} 2023-02-13 21:57:46,101 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325466100"}]},"ts":"9223372036854775807"} 2023-02-13 21:57:46,101 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325466100"}]},"ts":"9223372036854775807"} 2023-02-13 21:57:46,101 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325466100"}]},"ts":"9223372036854775807"} 2023-02-13 21:57:46,104 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-13 21:57:46,104 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 1f6dc79bf11da8444a0870d779d2fb4d, NAME => 'testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 251bc68b009792a14e42934b238a6375, NAME => 'testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 44e821dce55cb29b05989bca574cff62, NAME => 'testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 4e757d660af6a5aaa6e2c43f8b91311d, NAME => 'testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 676fb98d6cc62f0fdbda4ce59aad85cb, NAME => 'testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 992f98999ce7b2c13e6adb6eca28ff70, NAME => 'testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-13 21:57:46,104 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325466101"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325466101"}]},"ts":"1676325466101"} 2023-02-13 21:57:46,104 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325466101"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325466101"}]},"ts":"1676325466101"} 2023-02-13 21:57:46,104 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325466101"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325466101"}]},"ts":"1676325466101"} 2023-02-13 21:57:46,104 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325466101"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325466101"}]},"ts":"1676325466101"} 2023-02-13 21:57:46,105 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325466101"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325466101"}]},"ts":"1676325466101"} 2023-02-13 21:57:46,105 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325466101"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325466101"}]},"ts":"1676325466101"} 2023-02-13 21:57:46,107 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-13 21:57:46,107 INFO [PEWorker-4] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-13 21:57:46,107 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 1f6dc79bf11da8444a0870d779d2fb4d, NAME => 'testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 251bc68b009792a14e42934b238a6375, NAME => 'testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 44e821dce55cb29b05989bca574cff62, NAME => 'testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 4e757d660af6a5aaa6e2c43f8b91311d, NAME => 'testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 676fb98d6cc62f0fdbda4ce59aad85cb, NAME => 'testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 992f98999ce7b2c13e6adb6eca28ff70, NAME => 'testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-13 21:57:46,108 INFO [PEWorker-4] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb0-1676325454277 table=testRestoreSnapshot-1676325454277 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676325454277 completed! 2023-02-13 21:57:46,110 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=91, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676325454277 snapshot=name: "snaptb0-1676325454277" table: "testRestoreSnapshot-1676325454277" creation_time: 1676325458613 type: DISABLED version: 2 ttl: 0 ) in 74 msec 2023-02-13 21:57:46,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-02-13 21:57:46,298 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676325454277, procId: 91 completed 2023-02-13 21:57:46,298 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb0-1676325454277-1676325465744 2023-02-13 21:57:46,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(748): Client=jenkins//148.251.75.209 delete name: "hbase-failsafe-snaptb0-1676325454277-1676325465744" 2023-02-13 21:57:46,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb0-1676325454277-1676325465744 2023-02-13 21:57:46,307 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676325454277 2023-02-13 21:57:46,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.HMaster$10(2372): Client=jenkins//148.251.75.209 enable testRestoreSnapshot-1676325454277 2023-02-13 21:57:46,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=92, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:46,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-13 21:57:46,312 INFO [PEWorker-2] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676325454277 2023-02-13 21:57:46,312 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325466312"}]},"ts":"1676325466312"} 2023-02-13 21:57:46,314 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676325454277, state=ENABLING in hbase:meta 2023-02-13 21:57:46,316 DEBUG [PEWorker-2] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 21:57:46,319 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, ASSIGN}, {pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, ASSIGN}, {pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, ASSIGN}, {pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, ASSIGN}, {pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, ASSIGN}, {pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, ASSIGN}] 2023-02-13 21:57:46,321 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, ASSIGN 2023-02-13 21:57:46,322 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 21:57:46,323 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, ASSIGN 2023-02-13 21:57:46,323 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 21:57:46,330 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, ASSIGN 2023-02-13 21:57:46,330 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, ASSIGN 2023-02-13 21:57:46,330 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, ASSIGN 2023-02-13 21:57:46,330 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, ASSIGN 2023-02-13 21:57:46,331 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 21:57:46,331 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 21:57:46,331 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 21:57:46,331 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 21:57:46,472 DEBUG [jenkins-hbase20:46557] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-02-13 21:57:46,473 DEBUG [jenkins-hbase20:46557] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 21:57:46,473 DEBUG [jenkins-hbase20:46557] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 21:57:46,473 DEBUG [jenkins-hbase20:46557] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 21:57:46,473 DEBUG [jenkins-hbase20:46557] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 21:57:46,475 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=44e821dce55cb29b05989bca574cff62, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:46,475 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=676fb98d6cc62f0fdbda4ce59aad85cb, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:46,475 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=4e757d660af6a5aaa6e2c43f8b91311d, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:46,475 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=1f6dc79bf11da8444a0870d779d2fb4d, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:46,476 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325466475"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325466475"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325466475"}]},"ts":"1676325466475"} 2023-02-13 21:57:46,475 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=251bc68b009792a14e42934b238a6375, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:46,476 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325466475"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325466475"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325466475"}]},"ts":"1676325466475"} 2023-02-13 21:57:46,476 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325466475"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325466475"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325466475"}]},"ts":"1676325466475"} 2023-02-13 21:57:46,475 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325466475"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325466475"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325466475"}]},"ts":"1676325466475"} 2023-02-13 21:57:46,476 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325466475"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325466475"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325466475"}]},"ts":"1676325466475"} 2023-02-13 21:57:46,477 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=99, ppid=97, state=RUNNABLE; OpenRegionProcedure 4e757d660af6a5aaa6e2c43f8b91311d, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:57:46,478 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=100, ppid=93, state=RUNNABLE; OpenRegionProcedure 1f6dc79bf11da8444a0870d779d2fb4d, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:57:46,479 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=101, ppid=96, state=RUNNABLE; OpenRegionProcedure 676fb98d6cc62f0fdbda4ce59aad85cb, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:57:46,480 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=102, ppid=95, state=RUNNABLE; OpenRegionProcedure 44e821dce55cb29b05989bca574cff62, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:57:46,480 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=992f98999ce7b2c13e6adb6eca28ff70, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:46,480 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325466480"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325466480"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325466480"}]},"ts":"1676325466480"} 2023-02-13 21:57:46,480 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=103, ppid=98, state=RUNNABLE; OpenRegionProcedure 251bc68b009792a14e42934b238a6375, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:57:46,482 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=104, ppid=94, state=RUNNABLE; OpenRegionProcedure 992f98999ce7b2c13e6adb6eca28ff70, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:57:46,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-13 21:57:46,632 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:46,632 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:46,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 4e757d660af6a5aaa6e2c43f8b91311d, NAME => 'testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 21:57:46,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 676fb98d6cc62f0fdbda4ce59aad85cb, NAME => 'testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 21:57:46,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:46,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:46,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:46,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:46,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:46,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:46,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:46,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:46,635 INFO [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:46,635 INFO [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:46,635 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:46,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 251bc68b009792a14e42934b238a6375, NAME => 'testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 21:57:46,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:46,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:46,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:46,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:46,636 DEBUG [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf 2023-02-13 21:57:46,636 DEBUG [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf 2023-02-13 21:57:46,636 INFO [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 676fb98d6cc62f0fdbda4ce59aad85cb columnFamilyName cf 2023-02-13 21:57:46,637 DEBUG [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf 2023-02-13 21:57:46,638 DEBUG [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf 2023-02-13 21:57:46,638 INFO [StoreOpener-251bc68b009792a14e42934b238a6375-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:46,638 INFO [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4e757d660af6a5aaa6e2c43f8b91311d columnFamilyName cf 2023-02-13 21:57:46,643 DEBUG [StoreOpener-251bc68b009792a14e42934b238a6375-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf 2023-02-13 21:57:46,643 DEBUG [StoreOpener-251bc68b009792a14e42934b238a6375-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf 2023-02-13 21:57:46,645 INFO [StoreOpener-251bc68b009792a14e42934b238a6375-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 251bc68b009792a14e42934b238a6375 columnFamilyName cf 2023-02-13 21:57:46,650 DEBUG [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/99375a84534848d588fa45ac3699c454 2023-02-13 21:57:46,650 INFO [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] regionserver.HStore(310): Store=676fb98d6cc62f0fdbda4ce59aad85cb/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:46,651 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:46,652 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:46,652 DEBUG [StoreOpener-251bc68b009792a14e42934b238a6375-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/29f391c114814c4b96610425ae0e5de0 2023-02-13 21:57:46,652 DEBUG [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/81bb121c83f84b0997b6675992ad85a1 2023-02-13 21:57:46,653 INFO [StoreOpener-251bc68b009792a14e42934b238a6375-1] regionserver.HStore(310): Store=251bc68b009792a14e42934b238a6375/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:46,653 INFO [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] regionserver.HStore(310): Store=4e757d660af6a5aaa6e2c43f8b91311d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:46,654 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:46,654 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375 2023-02-13 21:57:46,654 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375 2023-02-13 21:57:46,655 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:46,655 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:46,657 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 676fb98d6cc62f0fdbda4ce59aad85cb; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66858939, jitterRate=-0.0037241727113723755}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:46,657 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 676fb98d6cc62f0fdbda4ce59aad85cb: Running coprocessor pre-open hook at 1676325466633Writing region info on filesystem at 1676325466633Initializing all the Stores at 1676325466634 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325466634Cleaning up temporary data from old regions at 1676325466652 (+18 ms)Cleaning up detritus from prior splits at 1676325466653 (+1 ms)Running coprocessor post-open hooks at 1676325466657 (+4 ms)Region opened successfully at 1676325466657 2023-02-13 21:57:46,658 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb., pid=101, masterSystemTime=1676325466629 2023-02-13 21:57:46,659 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:46,660 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:46,660 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:46,660 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1f6dc79bf11da8444a0870d779d2fb4d, NAME => 'testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 21:57:46,660 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=676fb98d6cc62f0fdbda4ce59aad85cb, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:46,660 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:46,660 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:46,660 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:46,660 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:46,660 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325466660"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325466660"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325466660"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325466660"}]},"ts":"1676325466660"} 2023-02-13 21:57:46,660 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:46,661 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:46,661 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 251bc68b009792a14e42934b238a6375; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63778197, jitterRate=-0.04963080585002899}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:46,661 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 4e757d660af6a5aaa6e2c43f8b91311d; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71983517, jitterRate=0.07263799011707306}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:46,661 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 251bc68b009792a14e42934b238a6375: Running coprocessor pre-open hook at 1676325466636Writing region info on filesystem at 1676325466636Initializing all the Stores at 1676325466637 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325466637Cleaning up temporary data from old regions at 1676325466655 (+18 ms)Cleaning up detritus from prior splits at 1676325466657 (+2 ms)Running coprocessor post-open hooks at 1676325466661 (+4 ms)Region opened successfully at 1676325466661 2023-02-13 21:57:46,661 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 4e757d660af6a5aaa6e2c43f8b91311d: Running coprocessor pre-open hook at 1676325466633Writing region info on filesystem at 1676325466633Initializing all the Stores at 1676325466634 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325466634Cleaning up temporary data from old regions at 1676325466655 (+21 ms)Cleaning up detritus from prior splits at 1676325466656 (+1 ms)Running coprocessor post-open hooks at 1676325466661 (+5 ms)Region opened successfully at 1676325466661 2023-02-13 21:57:46,662 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375., pid=103, masterSystemTime=1676325466632 2023-02-13 21:57:46,662 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d., pid=99, masterSystemTime=1676325466629 2023-02-13 21:57:46,664 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:46,664 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:46,664 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:46,665 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 44e821dce55cb29b05989bca574cff62, NAME => 'testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 21:57:46,665 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:46,665 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:46,665 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:46,665 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:46,666 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=251bc68b009792a14e42934b238a6375, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:46,666 INFO [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:46,666 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325466666"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325466666"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325466666"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325466666"}]},"ts":"1676325466666"} 2023-02-13 21:57:46,666 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:46,666 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:46,666 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:46,667 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 992f98999ce7b2c13e6adb6eca28ff70, NAME => 'testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 21:57:46,667 INFO [StoreOpener-44e821dce55cb29b05989bca574cff62-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:46,667 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:46,667 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=101, resume processing ppid=96 2023-02-13 21:57:46,667 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:46,667 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=4e757d660af6a5aaa6e2c43f8b91311d, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:46,667 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:46,667 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:46,667 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=101, ppid=96, state=SUCCESS; OpenRegionProcedure 676fb98d6cc62f0fdbda4ce59aad85cb, server=jenkins-hbase20.apache.org,45015,1676325451448 in 183 msec 2023-02-13 21:57:46,667 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325466667"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325466667"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325466667"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325466667"}]},"ts":"1676325466667"} 2023-02-13 21:57:46,669 DEBUG [StoreOpener-44e821dce55cb29b05989bca574cff62-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf 2023-02-13 21:57:46,669 DEBUG [StoreOpener-44e821dce55cb29b05989bca574cff62-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf 2023-02-13 21:57:46,669 INFO [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:46,669 DEBUG [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf 2023-02-13 21:57:46,669 DEBUG [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf 2023-02-13 21:57:46,670 INFO [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1f6dc79bf11da8444a0870d779d2fb4d columnFamilyName cf 2023-02-13 21:57:46,669 INFO [StoreOpener-44e821dce55cb29b05989bca574cff62-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 44e821dce55cb29b05989bca574cff62 columnFamilyName cf 2023-02-13 21:57:46,669 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=96, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, ASSIGN in 348 msec 2023-02-13 21:57:46,671 DEBUG [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf 2023-02-13 21:57:46,671 DEBUG [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf 2023-02-13 21:57:46,672 INFO [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 992f98999ce7b2c13e6adb6eca28ff70 columnFamilyName cf 2023-02-13 21:57:46,673 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=103, resume processing ppid=98 2023-02-13 21:57:46,673 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=103, ppid=98, state=SUCCESS; OpenRegionProcedure 251bc68b009792a14e42934b238a6375, server=jenkins-hbase20.apache.org,39523,1676325451480 in 189 msec 2023-02-13 21:57:46,674 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=99, resume processing ppid=97 2023-02-13 21:57:46,674 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=99, ppid=97, state=SUCCESS; OpenRegionProcedure 4e757d660af6a5aaa6e2c43f8b91311d, server=jenkins-hbase20.apache.org,36339,1676325451407 in 193 msec 2023-02-13 21:57:46,675 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=98, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, ASSIGN in 354 msec 2023-02-13 21:57:46,675 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=97, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, ASSIGN in 355 msec 2023-02-13 21:57:46,682 DEBUG [StoreOpener-44e821dce55cb29b05989bca574cff62-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/8795edd282b341fba51b1c197d77b4f9 2023-02-13 21:57:46,683 INFO [StoreOpener-44e821dce55cb29b05989bca574cff62-1] regionserver.HStore(310): Store=44e821dce55cb29b05989bca574cff62/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:46,683 DEBUG [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/ebd1b06b1c10407b903431c38f993f0f 2023-02-13 21:57:46,683 INFO [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] regionserver.HStore(310): Store=992f98999ce7b2c13e6adb6eca28ff70/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:46,684 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:46,684 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:46,686 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:46,686 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:46,687 DEBUG [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/3d85d09beeee48a5b67b39f0abb4165e 2023-02-13 21:57:46,687 INFO [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] regionserver.HStore(310): Store=1f6dc79bf11da8444a0870d779d2fb4d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:46,688 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:46,689 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:46,689 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:46,690 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:46,690 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 992f98999ce7b2c13e6adb6eca28ff70; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58949115, jitterRate=-0.12158973515033722}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:46,690 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 992f98999ce7b2c13e6adb6eca28ff70: Running coprocessor pre-open hook at 1676325466668Writing region info on filesystem at 1676325466668Initializing all the Stores at 1676325466669 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325466669Cleaning up temporary data from old regions at 1676325466686 (+17 ms)Cleaning up detritus from prior splits at 1676325466687 (+1 ms)Running coprocessor post-open hooks at 1676325466690 (+3 ms)Region opened successfully at 1676325466690 2023-02-13 21:57:46,691 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 44e821dce55cb29b05989bca574cff62; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74054520, jitterRate=0.10349833965301514}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:46,691 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 44e821dce55cb29b05989bca574cff62: Running coprocessor pre-open hook at 1676325466665Writing region info on filesystem at 1676325466665Initializing all the Stores at 1676325466666 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325466666Cleaning up temporary data from old regions at 1676325466686 (+20 ms)Cleaning up detritus from prior splits at 1676325466687 (+1 ms)Running coprocessor post-open hooks at 1676325466691 (+4 ms)Region opened successfully at 1676325466691 2023-02-13 21:57:46,691 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70., pid=104, masterSystemTime=1676325466629 2023-02-13 21:57:46,693 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62., pid=102, masterSystemTime=1676325466632 2023-02-13 21:57:46,694 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:46,694 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:46,694 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:46,695 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=992f98999ce7b2c13e6adb6eca28ff70, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:46,695 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325466694"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325466694"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325466694"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325466694"}]},"ts":"1676325466694"} 2023-02-13 21:57:46,696 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 1f6dc79bf11da8444a0870d779d2fb4d; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61657186, jitterRate=-0.081236332654953}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:46,696 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 1f6dc79bf11da8444a0870d779d2fb4d: Running coprocessor pre-open hook at 1676325466660Writing region info on filesystem at 1676325466660Initializing all the Stores at 1676325466662 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325466662Cleaning up temporary data from old regions at 1676325466690 (+28 ms)Cleaning up detritus from prior splits at 1676325466690Running coprocessor post-open hooks at 1676325466696 (+6 ms)Region opened successfully at 1676325466696 2023-02-13 21:57:46,699 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:46,699 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d., pid=100, masterSystemTime=1676325466629 2023-02-13 21:57:46,699 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:46,699 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=44e821dce55cb29b05989bca574cff62, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:46,699 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325466699"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325466699"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325466699"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325466699"}]},"ts":"1676325466699"} 2023-02-13 21:57:46,701 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:46,701 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:46,703 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=104, resume processing ppid=94 2023-02-13 21:57:46,703 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=1f6dc79bf11da8444a0870d779d2fb4d, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:46,703 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325466703"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325466703"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325466703"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325466703"}]},"ts":"1676325466703"} 2023-02-13 21:57:46,703 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=104, ppid=94, state=SUCCESS; OpenRegionProcedure 992f98999ce7b2c13e6adb6eca28ff70, server=jenkins-hbase20.apache.org,36339,1676325451407 in 217 msec 2023-02-13 21:57:46,705 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=102, resume processing ppid=95 2023-02-13 21:57:46,705 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=94, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, ASSIGN in 386 msec 2023-02-13 21:57:46,705 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=102, ppid=95, state=SUCCESS; OpenRegionProcedure 44e821dce55cb29b05989bca574cff62, server=jenkins-hbase20.apache.org,39523,1676325451480 in 221 msec 2023-02-13 21:57:46,706 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=95, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, ASSIGN in 386 msec 2023-02-13 21:57:46,707 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=100, resume processing ppid=93 2023-02-13 21:57:46,707 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=100, ppid=93, state=SUCCESS; OpenRegionProcedure 1f6dc79bf11da8444a0870d779d2fb4d, server=jenkins-hbase20.apache.org,45015,1676325451448 in 227 msec 2023-02-13 21:57:46,710 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=93, resume processing ppid=92 2023-02-13 21:57:46,710 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=93, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, ASSIGN in 390 msec 2023-02-13 21:57:46,711 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325466711"}]},"ts":"1676325466711"} 2023-02-13 21:57:46,713 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676325454277, state=ENABLED in hbase:meta 2023-02-13 21:57:46,715 INFO [PEWorker-5] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676325454277' was successfully enabled. 2023-02-13 21:57:46,719 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=92, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676325454277 in 408 msec 2023-02-13 21:57:47,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-13 21:57:47,064 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676325454277, procId: 92 completed 2023-02-13 21:57:47,065 DEBUG [Listener at localhost.localdomain/44075] zookeeper.ReadOnlyZKClient(139): Connect 0x555200b1 to 127.0.0.1:58580 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 21:57:47,070 DEBUG [Listener at localhost.localdomain/44075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ed71025, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 21:57:47,075 DEBUG [hconnection-0x58c4a1e2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 21:57:47,077 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:53058, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 21:57:47,085 DEBUG [hconnection-0x58c4a1e2-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 21:57:47,088 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49704, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 21:57:47,211 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 21:57:47,317 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 21:57:47,323 DEBUG [hconnection-0x58c4a1e2-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 21:57:47,326 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44922, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 21:57:47,451 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 21:57:47,637 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 21:57:47,686 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x36a3b58e to 127.0.0.1:58580 2023-02-13 21:57:47,686 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 21:57:47,797 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 21:57:48,994 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676325454277 2023-02-13 21:57:48,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.HMaster$11(2405): Client=jenkins//148.251.75.209 disable testRestoreSnapshot-1676325454277 2023-02-13 21:57:48,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=105, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:49,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-13 21:57:49,002 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325469002"}]},"ts":"1676325469002"} 2023-02-13 21:57:49,007 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676325454277, state=DISABLING in hbase:meta 2023-02-13 21:57:49,008 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676325454277 to state=DISABLING 2023-02-13 21:57:49,009 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, UNASSIGN}, {pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, UNASSIGN}, {pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, UNASSIGN}, {pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, UNASSIGN}, {pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, UNASSIGN}, {pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, UNASSIGN}] 2023-02-13 21:57:49,012 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, UNASSIGN 2023-02-13 21:57:49,012 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, UNASSIGN 2023-02-13 21:57:49,012 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, UNASSIGN 2023-02-13 21:57:49,012 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, UNASSIGN 2023-02-13 21:57:49,013 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, UNASSIGN 2023-02-13 21:57:49,013 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=1f6dc79bf11da8444a0870d779d2fb4d, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:49,013 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=44e821dce55cb29b05989bca574cff62, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:49,013 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325469013"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325469013"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325469013"}]},"ts":"1676325469013"} 2023-02-13 21:57:49,013 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325469013"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325469013"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325469013"}]},"ts":"1676325469013"} 2023-02-13 21:57:49,013 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=251bc68b009792a14e42934b238a6375, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:49,014 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325469013"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325469013"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325469013"}]},"ts":"1676325469013"} 2023-02-13 21:57:49,014 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=4e757d660af6a5aaa6e2c43f8b91311d, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:49,014 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325469014"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325469014"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325469014"}]},"ts":"1676325469014"} 2023-02-13 21:57:49,014 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=992f98999ce7b2c13e6adb6eca28ff70, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:49,014 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325469014"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325469014"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325469014"}]},"ts":"1676325469014"} 2023-02-13 21:57:49,015 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=112, ppid=108, state=RUNNABLE; CloseRegionProcedure 44e821dce55cb29b05989bca574cff62, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:57:49,020 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=113, ppid=106, state=RUNNABLE; CloseRegionProcedure 1f6dc79bf11da8444a0870d779d2fb4d, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:57:49,021 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=114, ppid=111, state=RUNNABLE; CloseRegionProcedure 251bc68b009792a14e42934b238a6375, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:57:49,022 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=115, ppid=110, state=RUNNABLE; CloseRegionProcedure 4e757d660af6a5aaa6e2c43f8b91311d, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:57:49,023 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=116, ppid=107, state=RUNNABLE; CloseRegionProcedure 992f98999ce7b2c13e6adb6eca28ff70, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:57:49,024 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, UNASSIGN 2023-02-13 21:57:49,026 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=676fb98d6cc62f0fdbda4ce59aad85cb, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:49,027 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325469026"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325469026"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325469026"}]},"ts":"1676325469026"} 2023-02-13 21:57:49,029 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=117, ppid=109, state=RUNNABLE; CloseRegionProcedure 676fb98d6cc62f0fdbda4ce59aad85cb, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:57:49,171 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:49,172 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 251bc68b009792a14e42934b238a6375, disabling compactions & flushes 2023-02-13 21:57:49,172 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:49,172 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:49,172 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. after waiting 0 ms 2023-02-13 21:57:49,172 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:49,176 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:49,179 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 676fb98d6cc62f0fdbda4ce59aad85cb, disabling compactions & flushes 2023-02-13 21:57:49,179 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:49,179 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:49,179 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. after waiting 0 ms 2023-02-13 21:57:49,179 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:49,179 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:49,179 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 992f98999ce7b2c13e6adb6eca28ff70, disabling compactions & flushes 2023-02-13 21:57:49,179 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:49,179 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:49,179 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. after waiting 0 ms 2023-02-13 21:57:49,179 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:49,189 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-13 21:57:49,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-13 21:57:49,194 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:49,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 251bc68b009792a14e42934b238a6375: Waiting for close lock at 1676325469171Running coprocessor pre-close hooks at 1676325469171Disabling compacts and flushes for region at 1676325469171Disabling writes for close at 1676325469172 (+1 ms)Writing region close event to WAL at 1676325469176 (+4 ms)Running coprocessor post-close hooks at 1676325469194 (+18 ms)Closed at 1676325469194 2023-02-13 21:57:49,194 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:49,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-13 21:57:49,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 676fb98d6cc62f0fdbda4ce59aad85cb: Waiting for close lock at 1676325469176Running coprocessor pre-close hooks at 1676325469176Disabling compacts and flushes for region at 1676325469176Disabling writes for close at 1676325469179 (+3 ms)Writing region close event to WAL at 1676325469187 (+8 ms)Running coprocessor post-close hooks at 1676325469194 (+7 ms)Closed at 1676325469194 2023-02-13 21:57:49,195 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:49,195 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 992f98999ce7b2c13e6adb6eca28ff70: Waiting for close lock at 1676325469179Running coprocessor pre-close hooks at 1676325469179Disabling compacts and flushes for region at 1676325469179Disabling writes for close at 1676325469179Writing region close event to WAL at 1676325469187 (+8 ms)Running coprocessor post-close hooks at 1676325469195 (+8 ms)Closed at 1676325469195 2023-02-13 21:57:49,196 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:49,196 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:49,196 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 44e821dce55cb29b05989bca574cff62, disabling compactions & flushes 2023-02-13 21:57:49,197 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:49,197 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:49,197 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. after waiting 0 ms 2023-02-13 21:57:49,197 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:49,197 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=251bc68b009792a14e42934b238a6375, regionState=CLOSED 2023-02-13 21:57:49,197 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325469197"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325469197"}]},"ts":"1676325469197"} 2023-02-13 21:57:49,197 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:49,197 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:49,197 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 1f6dc79bf11da8444a0870d779d2fb4d, disabling compactions & flushes 2023-02-13 21:57:49,197 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:49,197 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:49,198 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. after waiting 0 ms 2023-02-13 21:57:49,198 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:49,198 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=676fb98d6cc62f0fdbda4ce59aad85cb, regionState=CLOSED 2023-02-13 21:57:49,198 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325469198"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325469198"}]},"ts":"1676325469198"} 2023-02-13 21:57:49,199 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:49,199 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:49,199 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 4e757d660af6a5aaa6e2c43f8b91311d, disabling compactions & flushes 2023-02-13 21:57:49,199 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:49,199 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:49,199 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. after waiting 0 ms 2023-02-13 21:57:49,199 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:49,200 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=992f98999ce7b2c13e6adb6eca28ff70, regionState=CLOSED 2023-02-13 21:57:49,200 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325469200"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325469200"}]},"ts":"1676325469200"} 2023-02-13 21:57:49,201 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=114, resume processing ppid=111 2023-02-13 21:57:49,201 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=114, ppid=111, state=SUCCESS; CloseRegionProcedure 251bc68b009792a14e42934b238a6375, server=jenkins-hbase20.apache.org,39523,1676325451480 in 178 msec 2023-02-13 21:57:49,202 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=117, resume processing ppid=109 2023-02-13 21:57:49,202 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=117, ppid=109, state=SUCCESS; CloseRegionProcedure 676fb98d6cc62f0fdbda4ce59aad85cb, server=jenkins-hbase20.apache.org,45015,1676325451448 in 171 msec 2023-02-13 21:57:49,203 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=111, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, UNASSIGN in 192 msec 2023-02-13 21:57:49,204 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=116, resume processing ppid=107 2023-02-13 21:57:49,204 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=116, ppid=107, state=SUCCESS; CloseRegionProcedure 992f98999ce7b2c13e6adb6eca28ff70, server=jenkins-hbase20.apache.org,36339,1676325451407 in 178 msec 2023-02-13 21:57:49,204 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=109, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, UNASSIGN in 194 msec 2023-02-13 21:57:49,205 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=107, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, UNASSIGN in 195 msec 2023-02-13 21:57:49,207 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-13 21:57:49,208 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:49,208 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 44e821dce55cb29b05989bca574cff62: Waiting for close lock at 1676325469196Running coprocessor pre-close hooks at 1676325469196Disabling compacts and flushes for region at 1676325469196Disabling writes for close at 1676325469197 (+1 ms)Writing region close event to WAL at 1676325469198 (+1 ms)Running coprocessor post-close hooks at 1676325469208 (+10 ms)Closed at 1676325469208 2023-02-13 21:57:49,210 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:49,210 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-13 21:57:49,210 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=44e821dce55cb29b05989bca574cff62, regionState=CLOSED 2023-02-13 21:57:49,210 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-13 21:57:49,211 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325469210"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325469210"}]},"ts":"1676325469210"} 2023-02-13 21:57:49,212 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:49,212 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:49,212 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 4e757d660af6a5aaa6e2c43f8b91311d: Waiting for close lock at 1676325469199Running coprocessor pre-close hooks at 1676325469199Disabling compacts and flushes for region at 1676325469199Disabling writes for close at 1676325469199Writing region close event to WAL at 1676325469207 (+8 ms)Running coprocessor post-close hooks at 1676325469212 (+5 ms)Closed at 1676325469212 2023-02-13 21:57:49,212 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 1f6dc79bf11da8444a0870d779d2fb4d: Waiting for close lock at 1676325469197Running coprocessor pre-close hooks at 1676325469197Disabling compacts and flushes for region at 1676325469197Disabling writes for close at 1676325469198 (+1 ms)Writing region close event to WAL at 1676325469207 (+9 ms)Running coprocessor post-close hooks at 1676325469212 (+5 ms)Closed at 1676325469212 2023-02-13 21:57:49,214 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:49,214 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=1f6dc79bf11da8444a0870d779d2fb4d, regionState=CLOSED 2023-02-13 21:57:49,214 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325469214"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325469214"}]},"ts":"1676325469214"} 2023-02-13 21:57:49,214 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=112, resume processing ppid=108 2023-02-13 21:57:49,215 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=112, ppid=108, state=SUCCESS; CloseRegionProcedure 44e821dce55cb29b05989bca574cff62, server=jenkins-hbase20.apache.org,39523,1676325451480 in 198 msec 2023-02-13 21:57:49,215 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:49,215 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=4e757d660af6a5aaa6e2c43f8b91311d, regionState=CLOSED 2023-02-13 21:57:49,215 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325469215"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325469215"}]},"ts":"1676325469215"} 2023-02-13 21:57:49,215 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=108, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, UNASSIGN in 205 msec 2023-02-13 21:57:49,217 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=113, resume processing ppid=106 2023-02-13 21:57:49,217 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=113, ppid=106, state=SUCCESS; CloseRegionProcedure 1f6dc79bf11da8444a0870d779d2fb4d, server=jenkins-hbase20.apache.org,45015,1676325451448 in 199 msec 2023-02-13 21:57:49,218 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=115, resume processing ppid=110 2023-02-13 21:57:49,218 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=115, ppid=110, state=SUCCESS; CloseRegionProcedure 4e757d660af6a5aaa6e2c43f8b91311d, server=jenkins-hbase20.apache.org,36339,1676325451407 in 194 msec 2023-02-13 21:57:49,218 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=106, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, UNASSIGN in 208 msec 2023-02-13 21:57:49,219 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=110, resume processing ppid=105 2023-02-13 21:57:49,219 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=110, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, UNASSIGN in 209 msec 2023-02-13 21:57:49,220 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325469220"}]},"ts":"1676325469220"} 2023-02-13 21:57:49,221 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676325454277, state=DISABLED in hbase:meta 2023-02-13 21:57:49,222 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676325454277 to state=DISABLED 2023-02-13 21:57:49,223 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=105, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676325454277 in 227 msec 2023-02-13 21:57:49,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-13 21:57:49,252 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676325454277, procId: 105 completed 2023-02-13 21:57:49,264 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1676325454277-1676325469264 2023-02-13 21:57:49,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1580): Client=jenkins//148.251.75.209 snapshot request for:{ ss=hbase-failsafe-emptySnaptb-1676325454277-1676325469264 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 } 2023-02-13 21:57:49,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676325469265 (current time:1676325469265). 2023-02-13 21:57:49,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 21:57:49,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] zookeeper.ReadOnlyZKClient(139): Connect 0x531db8bd to 127.0.0.1:58580 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 21:57:49,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1fe47ea0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 21:57:49,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 21:57:49,279 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:53072, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 21:57:49,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x531db8bd to 127.0.0.1:58580 2023-02-13 21:57:49,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 21:57:49,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 21:57:49,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-emptySnaptb-1676325454277-1676325469264 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 } 2023-02-13 21:57:49,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676325454277, type=EXCLUSIVE 2023-02-13 21:57:49,285 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676325454277, type=EXCLUSIVE 2023-02-13 21:57:49,286 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=118, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676325454277, type=EXCLUSIVE; timeout=600000, timestamp=1676326069286 2023-02-13 21:57:49,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-emptySnaptb-1676325454277-1676325469264 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 } 2023-02-13 21:57:49,286 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1676325454277-1676325469264 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676325454277 2023-02-13 21:57:49,287 DEBUG [Listener at localhost.localdomain/44075] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-emptySnaptb-1676325454277-1676325469264 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 21:57:49,287 DEBUG [Listener at localhost.localdomain/44075] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 21:57:49,301 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1676325454277-1676325469264 table=testRestoreSnapshot-1676325454277 type=DISABLED ttl=0 } 2023-02-13 21:57:49,304 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:49,304 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:49,304 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:49,304 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:49,304 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:49,304 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:49,304 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:49,304 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:49,305 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:49,305 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:49,306 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/29f391c114814c4b96610425ae0e5de0] hfiles 2023-02-13 21:57:49,307 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/29f391c114814c4b96610425ae0e5de0 2023-02-13 21:57:49,307 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:49,307 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:49,307 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:49,307 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:49,309 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/81bb121c83f84b0997b6675992ad85a1] hfiles 2023-02-13 21:57:49,309 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/81bb121c83f84b0997b6675992ad85a1 2023-02-13 21:57:49,309 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/3d85d09beeee48a5b67b39f0abb4165e] hfiles 2023-02-13 21:57:49,309 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/3d85d09beeee48a5b67b39f0abb4165e 2023-02-13 21:57:49,313 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/8795edd282b341fba51b1c197d77b4f9] hfiles 2023-02-13 21:57:49,313 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/99375a84534848d588fa45ac3699c454] hfiles 2023-02-13 21:57:49,313 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/8795edd282b341fba51b1c197d77b4f9 2023-02-13 21:57:49,313 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/99375a84534848d588fa45ac3699c454 2023-02-13 21:57:49,313 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b] hfiles 2023-02-13 21:57:49,313 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e 2023-02-13 21:57:49,313 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 2023-02-13 21:57:49,313 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 2023-02-13 21:57:49,313 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e 2023-02-13 21:57:49,314 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a 2023-02-13 21:57:49,314 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/ebd1b06b1c10407b903431c38f993f0f] hfiles 2023-02-13 21:57:49,314 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b 2023-02-13 21:57:49,314 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/ebd1b06b1c10407b903431c38f993f0f 2023-02-13 21:57:49,367 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-emptySnaptb-1676325454277-1676325469264 table=testRestoreSnapshot-1676325454277 type=DISABLED ttl=0 } as finished. 2023-02-13 21:57:49,367 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-emptySnaptb-1676325454277-1676325469264 2023-02-13 21:57:49,368 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676325454277-1676325469264 2023-02-13 21:57:49,388 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676325454277-1676325469264 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/hbase-failsafe-emptySnaptb-1676325454277-1676325469264 2023-02-13 21:57:49,408 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-emptySnaptb-1676325454277-1676325469264 of table testRestoreSnapshot-1676325454277 completed 2023-02-13 21:57:49,408 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676325454277-1676325469264 2023-02-13 21:57:49,408 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676325454277-1676325469264 2023-02-13 21:57:49,408 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1676325454277-1676325469264 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676325454277 at 1676325469287Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1676325454277-1676325469264 table=testRestoreSnapshot-1676325454277 type=DISABLED ttl=0 } at 1676325469302 (+15 ms)Consolidate snapshot: hbase-failsafe-emptySnaptb-1676325454277-1676325469264 at 1676325469367 (+65 ms)Loading Region manifests for hbase-failsafe-emptySnaptb-1676325454277-1676325469264 at 1676325469367Writing data manifest for hbase-failsafe-emptySnaptb-1676325454277-1676325469264 at 1676325469371 (+4 ms)Verifying snapshot: hbase-failsafe-emptySnaptb-1676325454277-1676325469264 at 1676325469382 (+11 ms)Snapshot hbase-failsafe-emptySnaptb-1676325454277-1676325469264 of table testRestoreSnapshot-1676325454277 completed at 1676325469408 (+26 ms) 2023-02-13 21:57:49,410 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=118, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676325454277, type=EXCLUSIVE 2023-02-13 21:57:49,412 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=118, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676325454277, type=EXCLUSIVE in 127 msec 2023-02-13 21:57:49,537 DEBUG [Listener at localhost.localdomain/44075] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 21:57:49,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-emptySnaptb-1676325454277-1676325469264 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 } is done 2023-02-13 21:57:49,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-emptySnaptb-1676325454277-1676325469264 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 21:57:49,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=119, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676325454277 snapshot=name: "emptySnaptb-1676325454277" table: "testRestoreSnapshot-1676325454277" creation_time: 1676325455421 type: DISABLED version: 2 ttl: 0 ) 2023-02-13 21:57:49,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(880): Restore snapshot=emptySnaptb-1676325454277 as table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:49,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-13 21:57:49,567 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/.tabledesc/.tableinfo.0000000003 2023-02-13 21:57:49,568 DEBUG [PEWorker-5] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/.tabledesc/.tableinfo.0000000002 2023-02-13 21:57:49,569 INFO [PEWorker-5] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/.tabledesc/.tableinfo.0000000003 2023-02-13 21:57:49,570 INFO [PEWorker-5] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=emptySnaptb-1676325454277 table=testRestoreSnapshot-1676325454277 type=DISABLED ttl=0 } 2023-02-13 21:57:49,572 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "emptySnaptb-1676325454277" table: "testRestoreSnapshot-1676325454277" creation_time: 1676325455421 type: DISABLED version: 2 ttl: 0 2023-02-13 21:57:49,572 DEBUG [PEWorker-5] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277 2023-02-13 21:57:49,581 DEBUG [PEWorker-5] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:49,585 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e 2023-02-13 21:57:49,586 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 2023-02-13 21:57:49,587 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 2023-02-13 21:57:49,589 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e 2023-02-13 21:57:49,591 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a 2023-02-13 21:57:49,592 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b 2023-02-13 21:57:49,593 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:49,593 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:49,593 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:49,593 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:49,593 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:49,593 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:49,599 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/8795edd282b341fba51b1c197d77b4f9 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/8795edd282b341fba51b1c197d77b4f9 2023-02-13 21:57:49,599 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/99375a84534848d588fa45ac3699c454 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/99375a84534848d588fa45ac3699c454 2023-02-13 21:57:49,599 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/81bb121c83f84b0997b6675992ad85a1 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/81bb121c83f84b0997b6675992ad85a1 2023-02-13 21:57:49,600 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/3d85d09beeee48a5b67b39f0abb4165e to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/3d85d09beeee48a5b67b39f0abb4165e 2023-02-13 21:57:49,600 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/ebd1b06b1c10407b903431c38f993f0f to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/ebd1b06b1c10407b903431c38f993f0f 2023-02-13 21:57:49,600 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/29f391c114814c4b96610425ae0e5de0 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/29f391c114814c4b96610425ae0e5de0 2023-02-13 21:57:49,601 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "emptySnaptb-1676325454277" table: "testRestoreSnapshot-1676325454277" creation_time: 1676325455421 type: DISABLED version: 2 ttl: 0 2023-02-13 21:57:49,602 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325469602"}]},"ts":"9223372036854775807"} 2023-02-13 21:57:49,603 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325469602"}]},"ts":"9223372036854775807"} 2023-02-13 21:57:49,603 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325469602"}]},"ts":"9223372036854775807"} 2023-02-13 21:57:49,603 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325469602"}]},"ts":"9223372036854775807"} 2023-02-13 21:57:49,603 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325469602"}]},"ts":"9223372036854775807"} 2023-02-13 21:57:49,603 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325469602"}]},"ts":"9223372036854775807"} 2023-02-13 21:57:49,606 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-13 21:57:49,606 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 1f6dc79bf11da8444a0870d779d2fb4d, NAME => 'testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 251bc68b009792a14e42934b238a6375, NAME => 'testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 44e821dce55cb29b05989bca574cff62, NAME => 'testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 4e757d660af6a5aaa6e2c43f8b91311d, NAME => 'testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 676fb98d6cc62f0fdbda4ce59aad85cb, NAME => 'testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 992f98999ce7b2c13e6adb6eca28ff70, NAME => 'testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-13 21:57:49,607 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325469603"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325469603"}]},"ts":"1676325469603"} 2023-02-13 21:57:49,607 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325469603"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325469603"}]},"ts":"1676325469603"} 2023-02-13 21:57:49,607 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325469603"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325469603"}]},"ts":"1676325469603"} 2023-02-13 21:57:49,607 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325469603"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325469603"}]},"ts":"1676325469603"} 2023-02-13 21:57:49,607 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325469603"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325469603"}]},"ts":"1676325469603"} 2023-02-13 21:57:49,607 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325469603"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325469603"}]},"ts":"1676325469603"} 2023-02-13 21:57:49,610 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-13 21:57:49,610 INFO [PEWorker-5] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-13 21:57:49,610 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 1f6dc79bf11da8444a0870d779d2fb4d, NAME => 'testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 251bc68b009792a14e42934b238a6375, NAME => 'testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 44e821dce55cb29b05989bca574cff62, NAME => 'testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 4e757d660af6a5aaa6e2c43f8b91311d, NAME => 'testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 676fb98d6cc62f0fdbda4ce59aad85cb, NAME => 'testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 992f98999ce7b2c13e6adb6eca28ff70, NAME => 'testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-13 21:57:49,610 INFO [PEWorker-5] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=emptySnaptb-1676325454277 table=testRestoreSnapshot-1676325454277 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676325454277 completed! 2023-02-13 21:57:49,613 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=119, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676325454277 snapshot=name: "emptySnaptb-1676325454277" table: "testRestoreSnapshot-1676325454277" creation_time: 1676325455421 type: DISABLED version: 2 ttl: 0 ) in 62 msec 2023-02-13 21:57:49,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-13 21:57:49,805 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676325454277, procId: 119 completed 2023-02-13 21:57:49,805 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1676325454277-1676325469264 2023-02-13 21:57:49,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(748): Client=jenkins//148.251.75.209 delete name: "hbase-failsafe-emptySnaptb-1676325454277-1676325469264" 2023-02-13 21:57:49,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-emptySnaptb-1676325454277-1676325469264 2023-02-13 21:57:49,809 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676325454277 2023-02-13 21:57:49,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.HMaster$10(2372): Client=jenkins//148.251.75.209 enable testRestoreSnapshot-1676325454277 2023-02-13 21:57:49,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=120, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:49,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-13 21:57:49,813 INFO [PEWorker-4] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676325454277 2023-02-13 21:57:49,814 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325469813"}]},"ts":"1676325469813"} 2023-02-13 21:57:49,815 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676325454277, state=ENABLING in hbase:meta 2023-02-13 21:57:49,818 DEBUG [PEWorker-4] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 21:57:49,818 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, ASSIGN}, {pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, ASSIGN}, {pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, ASSIGN}, {pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, ASSIGN}, {pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, ASSIGN}, {pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, ASSIGN}] 2023-02-13 21:57:49,820 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, ASSIGN 2023-02-13 21:57:49,820 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, ASSIGN 2023-02-13 21:57:49,821 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, ASSIGN 2023-02-13 21:57:49,821 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, ASSIGN 2023-02-13 21:57:49,821 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 21:57:49,822 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 21:57:49,823 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 21:57:49,823 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, ASSIGN 2023-02-13 21:57:49,823 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 21:57:49,823 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, ASSIGN 2023-02-13 21:57:49,827 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 21:57:49,827 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 21:57:49,972 DEBUG [jenkins-hbase20:46557] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-02-13 21:57:49,972 DEBUG [jenkins-hbase20:46557] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 21:57:49,972 DEBUG [jenkins-hbase20:46557] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 21:57:49,972 DEBUG [jenkins-hbase20:46557] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 21:57:49,972 DEBUG [jenkins-hbase20:46557] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 21:57:49,981 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=676fb98d6cc62f0fdbda4ce59aad85cb, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:49,981 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=251bc68b009792a14e42934b238a6375, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:49,981 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=44e821dce55cb29b05989bca574cff62, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:49,981 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=4e757d660af6a5aaa6e2c43f8b91311d, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:49,981 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325469981"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325469981"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325469981"}]},"ts":"1676325469981"} 2023-02-13 21:57:49,982 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325469981"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325469981"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325469981"}]},"ts":"1676325469981"} 2023-02-13 21:57:49,981 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=1f6dc79bf11da8444a0870d779d2fb4d, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:49,981 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325469981"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325469981"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325469981"}]},"ts":"1676325469981"} 2023-02-13 21:57:49,981 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325469981"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325469981"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325469981"}]},"ts":"1676325469981"} 2023-02-13 21:57:49,982 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325469981"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325469981"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325469981"}]},"ts":"1676325469981"} 2023-02-13 21:57:49,985 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=127, ppid=124, state=RUNNABLE; OpenRegionProcedure 676fb98d6cc62f0fdbda4ce59aad85cb, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:57:49,985 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=128, ppid=125, state=RUNNABLE; OpenRegionProcedure 4e757d660af6a5aaa6e2c43f8b91311d, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:57:49,986 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=129, ppid=126, state=RUNNABLE; OpenRegionProcedure 251bc68b009792a14e42934b238a6375, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:57:49,988 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=130, ppid=123, state=RUNNABLE; OpenRegionProcedure 44e821dce55cb29b05989bca574cff62, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:57:49,989 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=131, ppid=121, state=RUNNABLE; OpenRegionProcedure 1f6dc79bf11da8444a0870d779d2fb4d, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:57:49,990 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=992f98999ce7b2c13e6adb6eca28ff70, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:49,990 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325469990"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325469990"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325469990"}]},"ts":"1676325469990"} 2023-02-13 21:57:49,991 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=132, ppid=122, state=RUNNABLE; OpenRegionProcedure 992f98999ce7b2c13e6adb6eca28ff70, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:57:50,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-13 21:57:50,139 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:50,140 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1f6dc79bf11da8444a0870d779d2fb4d, NAME => 'testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 21:57:50,140 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:50,140 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:50,140 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:50,140 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:50,141 INFO [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:50,141 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:50,141 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:50,142 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 251bc68b009792a14e42934b238a6375, NAME => 'testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 21:57:50,142 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 992f98999ce7b2c13e6adb6eca28ff70, NAME => 'testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 21:57:50,142 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:50,142 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:50,142 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:50,142 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:50,142 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:50,142 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:50,142 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:50,142 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:50,143 DEBUG [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf 2023-02-13 21:57:50,143 DEBUG [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf 2023-02-13 21:57:50,143 INFO [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1f6dc79bf11da8444a0870d779d2fb4d columnFamilyName cf 2023-02-13 21:57:50,144 INFO [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:50,144 INFO [StoreOpener-251bc68b009792a14e42934b238a6375-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:50,145 DEBUG [StoreOpener-251bc68b009792a14e42934b238a6375-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf 2023-02-13 21:57:50,145 DEBUG [StoreOpener-251bc68b009792a14e42934b238a6375-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf 2023-02-13 21:57:50,145 DEBUG [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf 2023-02-13 21:57:50,145 INFO [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] regionserver.HStore(310): Store=1f6dc79bf11da8444a0870d779d2fb4d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:50,145 DEBUG [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf 2023-02-13 21:57:50,145 INFO [StoreOpener-251bc68b009792a14e42934b238a6375-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 251bc68b009792a14e42934b238a6375 columnFamilyName cf 2023-02-13 21:57:50,145 INFO [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 992f98999ce7b2c13e6adb6eca28ff70 columnFamilyName cf 2023-02-13 21:57:50,146 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:50,146 INFO [StoreOpener-251bc68b009792a14e42934b238a6375-1] regionserver.HStore(310): Store=251bc68b009792a14e42934b238a6375/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:50,146 INFO [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] regionserver.HStore(310): Store=992f98999ce7b2c13e6adb6eca28ff70/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:50,147 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:50,147 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375 2023-02-13 21:57:50,147 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:50,148 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:50,148 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375 2023-02-13 21:57:50,149 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:50,150 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 1f6dc79bf11da8444a0870d779d2fb4d; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69283832, jitterRate=0.03240954875946045}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:50,150 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 1f6dc79bf11da8444a0870d779d2fb4d: Running coprocessor pre-open hook at 1676325470140Writing region info on filesystem at 1676325470140Initializing all the Stores at 1676325470141 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325470141Cleaning up temporary data from old regions at 1676325470147 (+6 ms)Cleaning up detritus from prior splits at 1676325470147Running coprocessor post-open hooks at 1676325470150 (+3 ms)Region opened successfully at 1676325470150 2023-02-13 21:57:50,150 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:50,150 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:50,150 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d., pid=131, masterSystemTime=1676325470137 2023-02-13 21:57:50,151 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 992f98999ce7b2c13e6adb6eca28ff70; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66106532, jitterRate=-0.014935910701751709}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:50,151 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 251bc68b009792a14e42934b238a6375; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59954533, jitterRate=-0.1066078394651413}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:50,151 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 992f98999ce7b2c13e6adb6eca28ff70: Running coprocessor pre-open hook at 1676325470142Writing region info on filesystem at 1676325470142Initializing all the Stores at 1676325470143 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325470143Cleaning up temporary data from old regions at 1676325470148 (+5 ms)Cleaning up detritus from prior splits at 1676325470148Running coprocessor post-open hooks at 1676325470151 (+3 ms)Region opened successfully at 1676325470151 2023-02-13 21:57:50,151 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 251bc68b009792a14e42934b238a6375: Running coprocessor pre-open hook at 1676325470143Writing region info on filesystem at 1676325470143Initializing all the Stores at 1676325470143Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325470143Cleaning up temporary data from old regions at 1676325470148 (+5 ms)Cleaning up detritus from prior splits at 1676325470148Running coprocessor post-open hooks at 1676325470151 (+3 ms)Region opened successfully at 1676325470151 2023-02-13 21:57:50,151 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70., pid=132, masterSystemTime=1676325470139 2023-02-13 21:57:50,151 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375., pid=129, masterSystemTime=1676325470139 2023-02-13 21:57:50,152 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:50,152 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:50,152 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:50,152 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 676fb98d6cc62f0fdbda4ce59aad85cb, NAME => 'testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 21:57:50,152 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=1f6dc79bf11da8444a0870d779d2fb4d, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:50,152 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325470152"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325470152"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325470152"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325470152"}]},"ts":"1676325470152"} 2023-02-13 21:57:50,152 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:50,153 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:50,153 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:50,153 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:50,153 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:50,153 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:50,153 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:50,153 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 4e757d660af6a5aaa6e2c43f8b91311d, NAME => 'testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 21:57:50,153 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:50,154 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:50,154 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:50,154 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:50,154 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=992f98999ce7b2c13e6adb6eca28ff70, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:50,154 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325470154"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325470154"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325470154"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325470154"}]},"ts":"1676325470154"} 2023-02-13 21:57:50,154 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:50,154 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:50,154 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:50,154 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=251bc68b009792a14e42934b238a6375, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:50,154 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 44e821dce55cb29b05989bca574cff62, NAME => 'testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 21:57:50,155 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325470154"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325470154"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325470154"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325470154"}]},"ts":"1676325470154"} 2023-02-13 21:57:50,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:50,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:50,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:50,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:50,156 INFO [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:50,156 INFO [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:50,156 INFO [StoreOpener-44e821dce55cb29b05989bca574cff62-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:50,157 DEBUG [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf 2023-02-13 21:57:50,157 DEBUG [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf 2023-02-13 21:57:50,157 DEBUG [StoreOpener-44e821dce55cb29b05989bca574cff62-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf 2023-02-13 21:57:50,158 DEBUG [StoreOpener-44e821dce55cb29b05989bca574cff62-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf 2023-02-13 21:57:50,158 DEBUG [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf 2023-02-13 21:57:50,158 INFO [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4e757d660af6a5aaa6e2c43f8b91311d columnFamilyName cf 2023-02-13 21:57:50,158 DEBUG [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf 2023-02-13 21:57:50,158 INFO [StoreOpener-44e821dce55cb29b05989bca574cff62-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 44e821dce55cb29b05989bca574cff62 columnFamilyName cf 2023-02-13 21:57:50,158 INFO [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 676fb98d6cc62f0fdbda4ce59aad85cb columnFamilyName cf 2023-02-13 21:57:50,158 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=131, resume processing ppid=121 2023-02-13 21:57:50,159 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=131, ppid=121, state=SUCCESS; OpenRegionProcedure 1f6dc79bf11da8444a0870d779d2fb4d, server=jenkins-hbase20.apache.org,45015,1676325451448 in 165 msec 2023-02-13 21:57:50,159 INFO [StoreOpener-44e821dce55cb29b05989bca574cff62-1] regionserver.HStore(310): Store=44e821dce55cb29b05989bca574cff62/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:50,159 INFO [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] regionserver.HStore(310): Store=4e757d660af6a5aaa6e2c43f8b91311d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:50,159 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=132, resume processing ppid=122 2023-02-13 21:57:50,159 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=132, ppid=122, state=SUCCESS; OpenRegionProcedure 992f98999ce7b2c13e6adb6eca28ff70, server=jenkins-hbase20.apache.org,36339,1676325451407 in 165 msec 2023-02-13 21:57:50,159 INFO [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] regionserver.HStore(310): Store=676fb98d6cc62f0fdbda4ce59aad85cb/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:50,160 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:50,160 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:50,160 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=121, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, ASSIGN in 340 msec 2023-02-13 21:57:50,160 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:50,161 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=129, resume processing ppid=126 2023-02-13 21:57:50,161 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=122, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, ASSIGN in 341 msec 2023-02-13 21:57:50,161 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=129, ppid=126, state=SUCCESS; OpenRegionProcedure 251bc68b009792a14e42934b238a6375, server=jenkins-hbase20.apache.org,39523,1676325451480 in 172 msec 2023-02-13 21:57:50,161 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:50,161 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:50,161 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:50,162 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=126, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, ASSIGN in 343 msec 2023-02-13 21:57:50,163 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:50,164 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 4e757d660af6a5aaa6e2c43f8b91311d; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65258254, jitterRate=-0.02757623791694641}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:50,164 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 4e757d660af6a5aaa6e2c43f8b91311d: Running coprocessor pre-open hook at 1676325470154Writing region info on filesystem at 1676325470154Initializing all the Stores at 1676325470155 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325470155Cleaning up temporary data from old regions at 1676325470161 (+6 ms)Cleaning up detritus from prior splits at 1676325470162 (+1 ms)Running coprocessor post-open hooks at 1676325470164 (+2 ms)Region opened successfully at 1676325470164 2023-02-13 21:57:50,164 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:50,164 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d., pid=128, masterSystemTime=1676325470139 2023-02-13 21:57:50,165 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 676fb98d6cc62f0fdbda4ce59aad85cb; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74139205, jitterRate=0.10476024448871613}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:50,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 676fb98d6cc62f0fdbda4ce59aad85cb: Running coprocessor pre-open hook at 1676325470153Writing region info on filesystem at 1676325470153Initializing all the Stores at 1676325470154 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325470155 (+1 ms)Cleaning up temporary data from old regions at 1676325470162 (+7 ms)Cleaning up detritus from prior splits at 1676325470162Running coprocessor post-open hooks at 1676325470165 (+3 ms)Region opened successfully at 1676325470165 2023-02-13 21:57:50,165 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb., pid=127, masterSystemTime=1676325470137 2023-02-13 21:57:50,166 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:50,166 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:50,166 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=4e757d660af6a5aaa6e2c43f8b91311d, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:50,166 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:50,166 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:50,166 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325470166"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325470166"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325470166"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325470166"}]},"ts":"1676325470166"} 2023-02-13 21:57:50,167 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=676fb98d6cc62f0fdbda4ce59aad85cb, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:50,167 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325470167"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325470167"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325470167"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325470167"}]},"ts":"1676325470167"} 2023-02-13 21:57:50,169 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=128, resume processing ppid=125 2023-02-13 21:57:50,170 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=128, ppid=125, state=SUCCESS; OpenRegionProcedure 4e757d660af6a5aaa6e2c43f8b91311d, server=jenkins-hbase20.apache.org,36339,1676325451407 in 183 msec 2023-02-13 21:57:50,171 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=127, resume processing ppid=124 2023-02-13 21:57:50,171 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=127, ppid=124, state=SUCCESS; OpenRegionProcedure 676fb98d6cc62f0fdbda4ce59aad85cb, server=jenkins-hbase20.apache.org,45015,1676325451448 in 184 msec 2023-02-13 21:57:50,171 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=125, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, ASSIGN in 352 msec 2023-02-13 21:57:50,172 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:50,172 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=124, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, ASSIGN in 353 msec 2023-02-13 21:57:50,172 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 44e821dce55cb29b05989bca574cff62; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62551175, jitterRate=-0.06791485846042633}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:50,172 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 44e821dce55cb29b05989bca574cff62: Running coprocessor pre-open hook at 1676325470155Writing region info on filesystem at 1676325470155Initializing all the Stores at 1676325470156 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325470156Cleaning up temporary data from old regions at 1676325470169 (+13 ms)Cleaning up detritus from prior splits at 1676325470170 (+1 ms)Running coprocessor post-open hooks at 1676325470172 (+2 ms)Region opened successfully at 1676325470172 2023-02-13 21:57:50,173 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62., pid=130, masterSystemTime=1676325470139 2023-02-13 21:57:50,174 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:50,174 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:50,175 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=44e821dce55cb29b05989bca574cff62, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:50,175 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325470174"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325470174"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325470174"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325470174"}]},"ts":"1676325470174"} 2023-02-13 21:57:50,178 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=130, resume processing ppid=123 2023-02-13 21:57:50,178 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=130, ppid=123, state=SUCCESS; OpenRegionProcedure 44e821dce55cb29b05989bca574cff62, server=jenkins-hbase20.apache.org,39523,1676325451480 in 188 msec 2023-02-13 21:57:50,179 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=123, resume processing ppid=120 2023-02-13 21:57:50,179 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=123, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, ASSIGN in 360 msec 2023-02-13 21:57:50,180 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325470180"}]},"ts":"1676325470180"} 2023-02-13 21:57:50,181 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676325454277, state=ENABLED in hbase:meta 2023-02-13 21:57:50,182 INFO [PEWorker-3] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676325454277' was successfully enabled. 2023-02-13 21:57:50,184 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=120, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676325454277 in 373 msec 2023-02-13 21:57:50,485 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 21:57:50,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-13 21:57:50,565 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676325454277, procId: 120 completed 2023-02-13 21:57:50,566 DEBUG [Listener at localhost.localdomain/44075] zookeeper.ReadOnlyZKClient(139): Connect 0x356c47e3 to 127.0.0.1:58580 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 21:57:50,570 DEBUG [Listener at localhost.localdomain/44075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@585fafe4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 21:57:50,572 DEBUG [hconnection-0x7c944828-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 21:57:50,574 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:53084, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 21:57:50,578 DEBUG [hconnection-0x7c944828-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 21:57:50,580 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49706, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 21:57:50,581 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 21:57:50,584 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 21:57:50,586 DEBUG [hconnection-0x7c944828-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 21:57:50,590 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44924, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 21:57:50,591 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 21:57:50,594 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 21:57:50,596 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 21:57:50,602 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676325454277 2023-02-13 21:57:50,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.HMaster$11(2405): Client=jenkins//148.251.75.209 disable testRestoreSnapshot-1676325454277 2023-02-13 21:57:50,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=133, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:50,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-13 21:57:50,606 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325470606"}]},"ts":"1676325470606"} 2023-02-13 21:57:50,607 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676325454277, state=DISABLING in hbase:meta 2023-02-13 21:57:50,608 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676325454277 to state=DISABLING 2023-02-13 21:57:50,609 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, UNASSIGN}, {pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, UNASSIGN}, {pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, UNASSIGN}, {pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, UNASSIGN}, {pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, UNASSIGN}, {pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, UNASSIGN}] 2023-02-13 21:57:50,611 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, UNASSIGN 2023-02-13 21:57:50,611 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, UNASSIGN 2023-02-13 21:57:50,612 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, UNASSIGN 2023-02-13 21:57:50,612 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, UNASSIGN 2023-02-13 21:57:50,612 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, UNASSIGN 2023-02-13 21:57:50,612 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=4e757d660af6a5aaa6e2c43f8b91311d, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:50,612 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325470612"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325470612"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325470612"}]},"ts":"1676325470612"} 2023-02-13 21:57:50,613 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=676fb98d6cc62f0fdbda4ce59aad85cb, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:50,613 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=44e821dce55cb29b05989bca574cff62, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:50,613 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=992f98999ce7b2c13e6adb6eca28ff70, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:50,613 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325470613"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325470613"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325470613"}]},"ts":"1676325470613"} 2023-02-13 21:57:50,613 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325470613"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325470613"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325470613"}]},"ts":"1676325470613"} 2023-02-13 21:57:50,613 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=251bc68b009792a14e42934b238a6375, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:50,613 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325470613"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325470613"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325470613"}]},"ts":"1676325470613"} 2023-02-13 21:57:50,613 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325470613"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325470613"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325470613"}]},"ts":"1676325470613"} 2023-02-13 21:57:50,614 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=140, ppid=136, state=RUNNABLE; CloseRegionProcedure 44e821dce55cb29b05989bca574cff62, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:57:50,615 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=141, ppid=138, state=RUNNABLE; CloseRegionProcedure 4e757d660af6a5aaa6e2c43f8b91311d, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:57:50,616 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=142, ppid=135, state=RUNNABLE; CloseRegionProcedure 992f98999ce7b2c13e6adb6eca28ff70, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:57:50,616 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=143, ppid=137, state=RUNNABLE; CloseRegionProcedure 676fb98d6cc62f0fdbda4ce59aad85cb, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:57:50,618 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, UNASSIGN 2023-02-13 21:57:50,619 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=1f6dc79bf11da8444a0870d779d2fb4d, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:50,619 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325470619"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325470619"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325470619"}]},"ts":"1676325470619"} 2023-02-13 21:57:50,622 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=144, ppid=139, state=RUNNABLE; CloseRegionProcedure 251bc68b009792a14e42934b238a6375, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:57:50,623 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=145, ppid=134, state=RUNNABLE; CloseRegionProcedure 1f6dc79bf11da8444a0870d779d2fb4d, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:57:50,768 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:50,768 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:50,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 4e757d660af6a5aaa6e2c43f8b91311d, disabling compactions & flushes 2023-02-13 21:57:50,769 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:50,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:50,770 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 44e821dce55cb29b05989bca574cff62, disabling compactions & flushes 2023-02-13 21:57:50,770 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:50,770 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:50,770 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. after waiting 0 ms 2023-02-13 21:57:50,770 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:50,770 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. after waiting 0 ms 2023-02-13 21:57:50,770 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:50,770 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:50,773 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 1f6dc79bf11da8444a0870d779d2fb4d, disabling compactions & flushes 2023-02-13 21:57:50,773 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:50,774 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:50,774 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. after waiting 0 ms 2023-02-13 21:57:50,774 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:50,777 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-13 21:57:50,777 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-13 21:57:50,778 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-13 21:57:50,779 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:50,779 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:50,779 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 4e757d660af6a5aaa6e2c43f8b91311d: Waiting for close lock at 1676325470768Running coprocessor pre-close hooks at 1676325470768Disabling compacts and flushes for region at 1676325470768Disabling writes for close at 1676325470770 (+2 ms)Writing region close event to WAL at 1676325470773 (+3 ms)Running coprocessor post-close hooks at 1676325470778 (+5 ms)Closed at 1676325470779 (+1 ms) 2023-02-13 21:57:50,779 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 44e821dce55cb29b05989bca574cff62: Waiting for close lock at 1676325470769Running coprocessor pre-close hooks at 1676325470769Disabling compacts and flushes for region at 1676325470769Disabling writes for close at 1676325470770 (+1 ms)Writing region close event to WAL at 1676325470773 (+3 ms)Running coprocessor post-close hooks at 1676325470778 (+5 ms)Closed at 1676325470779 (+1 ms) 2023-02-13 21:57:50,780 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:50,780 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 1f6dc79bf11da8444a0870d779d2fb4d: Waiting for close lock at 1676325470770Running coprocessor pre-close hooks at 1676325470770Disabling compacts and flushes for region at 1676325470770Disabling writes for close at 1676325470774 (+4 ms)Writing region close event to WAL at 1676325470774Running coprocessor post-close hooks at 1676325470779 (+5 ms)Closed at 1676325470780 (+1 ms) 2023-02-13 21:57:50,781 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:50,781 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:50,785 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 992f98999ce7b2c13e6adb6eca28ff70, disabling compactions & flushes 2023-02-13 21:57:50,786 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:50,786 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:50,786 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. after waiting 0 ms 2023-02-13 21:57:50,786 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:50,787 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=4e757d660af6a5aaa6e2c43f8b91311d, regionState=CLOSED 2023-02-13 21:57:50,787 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325470787"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325470787"}]},"ts":"1676325470787"} 2023-02-13 21:57:50,787 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:50,787 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:50,789 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 676fb98d6cc62f0fdbda4ce59aad85cb, disabling compactions & flushes 2023-02-13 21:57:50,789 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:50,789 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:50,789 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. after waiting 0 ms 2023-02-13 21:57:50,789 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:50,789 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=1f6dc79bf11da8444a0870d779d2fb4d, regionState=CLOSED 2023-02-13 21:57:50,789 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325470789"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325470789"}]},"ts":"1676325470789"} 2023-02-13 21:57:50,789 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:50,789 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:50,790 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 251bc68b009792a14e42934b238a6375, disabling compactions & flushes 2023-02-13 21:57:50,790 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:50,791 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:50,791 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. after waiting 0 ms 2023-02-13 21:57:50,791 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:50,791 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=44e821dce55cb29b05989bca574cff62, regionState=CLOSED 2023-02-13 21:57:50,791 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325470791"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325470791"}]},"ts":"1676325470791"} 2023-02-13 21:57:50,794 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=141, resume processing ppid=138 2023-02-13 21:57:50,794 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=141, ppid=138, state=SUCCESS; CloseRegionProcedure 4e757d660af6a5aaa6e2c43f8b91311d, server=jenkins-hbase20.apache.org,36339,1676325451407 in 176 msec 2023-02-13 21:57:50,795 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=145, resume processing ppid=134 2023-02-13 21:57:50,795 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=138, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, UNASSIGN in 185 msec 2023-02-13 21:57:50,795 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=145, ppid=134, state=SUCCESS; CloseRegionProcedure 1f6dc79bf11da8444a0870d779d2fb4d, server=jenkins-hbase20.apache.org,45015,1676325451448 in 169 msec 2023-02-13 21:57:50,795 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=140, resume processing ppid=136 2023-02-13 21:57:50,795 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=140, ppid=136, state=SUCCESS; CloseRegionProcedure 44e821dce55cb29b05989bca574cff62, server=jenkins-hbase20.apache.org,39523,1676325451480 in 179 msec 2023-02-13 21:57:50,796 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-13 21:57:50,796 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-13 21:57:50,797 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:50,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 676fb98d6cc62f0fdbda4ce59aad85cb: Waiting for close lock at 1676325470788Running coprocessor pre-close hooks at 1676325470788Disabling compacts and flushes for region at 1676325470788Disabling writes for close at 1676325470789 (+1 ms)Writing region close event to WAL at 1676325470789Running coprocessor post-close hooks at 1676325470796 (+7 ms)Closed at 1676325470797 (+1 ms) 2023-02-13 21:57:50,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-13 21:57:50,798 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:50,798 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 251bc68b009792a14e42934b238a6375: Waiting for close lock at 1676325470789Running coprocessor pre-close hooks at 1676325470789Disabling compacts and flushes for region at 1676325470789Disabling writes for close at 1676325470791 (+2 ms)Writing region close event to WAL at 1676325470791Running coprocessor post-close hooks at 1676325470797 (+6 ms)Closed at 1676325470798 (+1 ms) 2023-02-13 21:57:50,800 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:50,800 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=134, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, UNASSIGN in 186 msec 2023-02-13 21:57:50,800 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 992f98999ce7b2c13e6adb6eca28ff70: Waiting for close lock at 1676325470781Running coprocessor pre-close hooks at 1676325470781Disabling compacts and flushes for region at 1676325470781Disabling writes for close at 1676325470786 (+5 ms)Writing region close event to WAL at 1676325470787 (+1 ms)Running coprocessor post-close hooks at 1676325470799 (+12 ms)Closed at 1676325470800 (+1 ms) 2023-02-13 21:57:50,800 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=136, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, UNASSIGN in 186 msec 2023-02-13 21:57:50,800 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:50,801 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=676fb98d6cc62f0fdbda4ce59aad85cb, regionState=CLOSED 2023-02-13 21:57:50,801 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:50,801 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325470801"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325470801"}]},"ts":"1676325470801"} 2023-02-13 21:57:50,801 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=251bc68b009792a14e42934b238a6375, regionState=CLOSED 2023-02-13 21:57:50,801 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325470801"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325470801"}]},"ts":"1676325470801"} 2023-02-13 21:57:50,802 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:50,802 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=992f98999ce7b2c13e6adb6eca28ff70, regionState=CLOSED 2023-02-13 21:57:50,802 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325470802"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325470802"}]},"ts":"1676325470802"} 2023-02-13 21:57:50,806 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=143, resume processing ppid=137 2023-02-13 21:57:50,806 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=143, ppid=137, state=SUCCESS; CloseRegionProcedure 676fb98d6cc62f0fdbda4ce59aad85cb, server=jenkins-hbase20.apache.org,45015,1676325451448 in 186 msec 2023-02-13 21:57:50,812 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=144, resume processing ppid=139 2023-02-13 21:57:50,813 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=137, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, UNASSIGN in 197 msec 2023-02-13 21:57:50,813 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=144, ppid=139, state=SUCCESS; CloseRegionProcedure 251bc68b009792a14e42934b238a6375, server=jenkins-hbase20.apache.org,39523,1676325451480 in 183 msec 2023-02-13 21:57:50,813 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=142, resume processing ppid=135 2023-02-13 21:57:50,813 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=142, ppid=135, state=SUCCESS; CloseRegionProcedure 992f98999ce7b2c13e6adb6eca28ff70, server=jenkins-hbase20.apache.org,36339,1676325451407 in 189 msec 2023-02-13 21:57:50,815 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=135, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, UNASSIGN in 204 msec 2023-02-13 21:57:50,815 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=139, resume processing ppid=133 2023-02-13 21:57:50,815 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=139, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, UNASSIGN in 204 msec 2023-02-13 21:57:50,816 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325470816"}]},"ts":"1676325470816"} 2023-02-13 21:57:50,817 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676325454277, state=DISABLED in hbase:meta 2023-02-13 21:57:50,818 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676325454277 to state=DISABLED 2023-02-13 21:57:50,820 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=133, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676325454277 in 216 msec 2023-02-13 21:57:50,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-13 21:57:50,857 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676325454277, procId: 133 completed 2023-02-13 21:57:50,866 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb1-1676325454277-1676325470866 2023-02-13 21:57:50,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1580): Client=jenkins//148.251.75.209 snapshot request for:{ ss=hbase-failsafe-snaptb1-1676325454277-1676325470866 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 } 2023-02-13 21:57:50,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676325470867 (current time:1676325470867). 2023-02-13 21:57:50,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 21:57:50,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] zookeeper.ReadOnlyZKClient(139): Connect 0x0f08566c to 127.0.0.1:58580 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 21:57:50,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@16fa22a4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 21:57:50,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 21:57:50,874 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:53096, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 21:57:50,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0f08566c to 127.0.0.1:58580 2023-02-13 21:57:50,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 21:57:50,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 21:57:50,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb1-1676325454277-1676325470866 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 } 2023-02-13 21:57:50,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676325454277, type=EXCLUSIVE 2023-02-13 21:57:50,878 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676325454277, type=EXCLUSIVE 2023-02-13 21:57:50,879 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=146, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676325454277, type=EXCLUSIVE; timeout=600000, timestamp=1676326070879 2023-02-13 21:57:50,879 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb1-1676325454277-1676325470866 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676325454277 2023-02-13 21:57:50,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb1-1676325454277-1676325470866 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 } 2023-02-13 21:57:50,879 DEBUG [Listener at localhost.localdomain/44075] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb1-1676325454277-1676325470866 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 21:57:50,879 DEBUG [Listener at localhost.localdomain/44075] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 21:57:50,891 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1676325454277-1676325470866 table=testRestoreSnapshot-1676325454277 type=DISABLED ttl=0 } 2023-02-13 21:57:50,895 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:50,895 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:50,895 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:50,895 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:50,895 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:50,895 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:50,895 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:50,895 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:50,896 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:50,896 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:50,896 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:50,896 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:50,896 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 21:57:50,896 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:57:50,896 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:57:50,896 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 21:57:50,897 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 21:57:50,897 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 21:57:50,897 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 21:57:50,897 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 21:57:50,903 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 21:57:51,130 DEBUG [Listener at localhost.localdomain/44075] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 21:57:51,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb1-1676325454277-1676325470866 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 } is done 2023-02-13 21:57:51,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(404): Snapshoting '{ ss=hbase-failsafe-snaptb1-1676325454277-1676325470866 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 }' is still in progress! 2023-02-13 21:57:51,131 DEBUG [Listener at localhost.localdomain/44075] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-13 21:57:51,347 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb1-1676325454277-1676325470866 table=testRestoreSnapshot-1676325454277 type=DISABLED ttl=0 } as finished. 2023-02-13 21:57:51,347 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb1-1676325454277-1676325470866 2023-02-13 21:57:51,348 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676325454277-1676325470866 2023-02-13 21:57:51,380 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676325454277-1676325470866 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/hbase-failsafe-snaptb1-1676325454277-1676325470866 2023-02-13 21:57:51,398 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb1-1676325454277-1676325470866 of table testRestoreSnapshot-1676325454277 completed 2023-02-13 21:57:51,398 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676325454277-1676325470866 2023-02-13 21:57:51,399 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676325454277-1676325470866 2023-02-13 21:57:51,399 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb1-1676325454277-1676325470866 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676325454277 at 1676325470879Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1676325454277-1676325470866 table=testRestoreSnapshot-1676325454277 type=DISABLED ttl=0 } at 1676325470891 (+12 ms)Consolidate snapshot: hbase-failsafe-snaptb1-1676325454277-1676325470866 at 1676325471347 (+456 ms)Loading Region manifests for hbase-failsafe-snaptb1-1676325454277-1676325470866 at 1676325471348 (+1 ms)Writing data manifest for hbase-failsafe-snaptb1-1676325454277-1676325470866 at 1676325471361 (+13 ms)Verifying snapshot: hbase-failsafe-snaptb1-1676325454277-1676325470866 at 1676325471374 (+13 ms)Snapshot hbase-failsafe-snaptb1-1676325454277-1676325470866 of table testRestoreSnapshot-1676325454277 completed at 1676325471398 (+24 ms) 2023-02-13 21:57:51,400 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=146, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676325454277, type=EXCLUSIVE 2023-02-13 21:57:51,402 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=146, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676325454277, type=EXCLUSIVE in 523 msec 2023-02-13 21:57:51,631 DEBUG [Listener at localhost.localdomain/44075] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 21:57:51,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb1-1676325454277-1676325470866 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 } is done 2023-02-13 21:57:51,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb1-1676325454277-1676325470866 table=testRestoreSnapshot-1676325454277 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 21:57:51,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=147, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676325454277 snapshot=name: "snaptb1-1676325454277" table: "testRestoreSnapshot-1676325454277" creation_time: 1676325465457 type: DISABLED version: 2 ttl: 0 ) 2023-02-13 21:57:51,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(880): Restore snapshot=snaptb1-1676325454277 as table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:51,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-02-13 21:57:51,660 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/.tabledesc/.tableinfo.0000000004 2023-02-13 21:57:51,661 DEBUG [PEWorker-1] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/.tabledesc/.tableinfo.0000000003 2023-02-13 21:57:51,661 INFO [PEWorker-1] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/.tabledesc/.tableinfo.0000000004 2023-02-13 21:57:51,662 INFO [PEWorker-1] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb1-1676325454277 table=testRestoreSnapshot-1676325454277 type=DISABLED ttl=0 } 2023-02-13 21:57:51,664 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1676325454277" table: "testRestoreSnapshot-1676325454277" creation_time: 1676325465457 type: DISABLED version: 2 ttl: 0 2023-02-13 21:57:51,664 DEBUG [PEWorker-1] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277 2023-02-13 21:57:51,672 DEBUG [PEWorker-1] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:51,673 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e of snapshot=snaptb1-1676325454277 to region=6a0489fa0c27bf9bed60f657753c7881 table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:51,677 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 of snapshot=snaptb1-1676325454277 to region=6a0489fa0c27bf9bed60f657753c7881 table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:51,680 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 of snapshot=snaptb1-1676325454277 to region=6a0489fa0c27bf9bed60f657753c7881 table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:51,683 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 of snapshot=snaptb1-1676325454277 to region=6a0489fa0c27bf9bed60f657753c7881 table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:51,687 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 of snapshot=snaptb1-1676325454277 to region=6a0489fa0c27bf9bed60f657753c7881 table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:51,690 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 of snapshot=snaptb1-1676325454277 to region=6a0489fa0c27bf9bed60f657753c7881 table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:51,693 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e of snapshot=snaptb1-1676325454277 to region=6a0489fa0c27bf9bed60f657753c7881 table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:51,696 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 of snapshot=snaptb1-1676325454277 to region=6a0489fa0c27bf9bed60f657753c7881 table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:51,700 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a of snapshot=snaptb1-1676325454277 to region=6a0489fa0c27bf9bed60f657753c7881 table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:51,716 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 of snapshot=snaptb1-1676325454277 to region=6a0489fa0c27bf9bed60f657753c7881 table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:51,723 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 of snapshot=snaptb1-1676325454277 to region=6a0489fa0c27bf9bed60f657753c7881 table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:51,727 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b of snapshot=snaptb1-1676325454277 to region=6a0489fa0c27bf9bed60f657753c7881 table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:51,731 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:51,731 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:51,731 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:51,731 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:51,731 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:51,731 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:51,735 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 3d85d09beeee48a5b67b39f0abb4165e of snapshot=snaptb1-1676325454277 to region=1f6dc79bf11da8444a0870d779d2fb4d table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:51,735 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 8795edd282b341fba51b1c197d77b4f9 of snapshot=snaptb1-1676325454277 to region=44e821dce55cb29b05989bca574cff62 table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:51,736 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 136600ed36234d64b213b50e0d6f02a9 of snapshot=snaptb1-1676325454277 to region=251bc68b009792a14e42934b238a6375 table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:51,737 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 24e696c302c64bd2bfbbbddca37e4305 of snapshot=snaptb1-1676325454277 to region=676fb98d6cc62f0fdbda4ce59aad85cb table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:51,739 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 08e8b394e04742929b62615c32c8ed4c of snapshot=snaptb1-1676325454277 to region=4e757d660af6a5aaa6e2c43f8b91311d table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:51,746 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink ecd3475741414bf0a2161d2b27bc209a of snapshot=snaptb1-1676325454277 to region=1f6dc79bf11da8444a0870d779d2fb4d table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:51,746 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 46bad1cc5702482388220a80e425cbe0 of snapshot=snaptb1-1676325454277 to region=992f98999ce7b2c13e6adb6eca28ff70 table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:51,751 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink f9ada1ebb3d0446a9743d17910965899 of snapshot=snaptb1-1676325454277 to region=44e821dce55cb29b05989bca574cff62 table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:51,751 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 81bb121c83f84b0997b6675992ad85a1 of snapshot=snaptb1-1676325454277 to region=4e757d660af6a5aaa6e2c43f8b91311d table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:51,762 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink ebd1b06b1c10407b903431c38f993f0f of snapshot=snaptb1-1676325454277 to region=992f98999ce7b2c13e6adb6eca28ff70 table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:51,762 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 29f391c114814c4b96610425ae0e5de0 of snapshot=snaptb1-1676325454277 to region=251bc68b009792a14e42934b238a6375 table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:51,772 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 99375a84534848d588fa45ac3699c454 of snapshot=snaptb1-1676325454277 to region=676fb98d6cc62f0fdbda4ce59aad85cb table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:51,776 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1676325454277" table: "testRestoreSnapshot-1676325454277" creation_time: 1676325465457 type: DISABLED version: 2 ttl: 0 2023-02-13 21:57:51,779 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325471779"}]},"ts":"9223372036854775807"} 2023-02-13 21:57:51,779 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325471779"}]},"ts":"9223372036854775807"} 2023-02-13 21:57:51,779 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325471779"}]},"ts":"9223372036854775807"} 2023-02-13 21:57:51,779 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325471779"}]},"ts":"9223372036854775807"} 2023-02-13 21:57:51,779 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325471779"}]},"ts":"9223372036854775807"} 2023-02-13 21:57:51,779 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325471779"}]},"ts":"9223372036854775807"} 2023-02-13 21:57:51,781 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-13 21:57:51,782 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 1f6dc79bf11da8444a0870d779d2fb4d, NAME => 'testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 251bc68b009792a14e42934b238a6375, NAME => 'testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 44e821dce55cb29b05989bca574cff62, NAME => 'testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 4e757d660af6a5aaa6e2c43f8b91311d, NAME => 'testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 676fb98d6cc62f0fdbda4ce59aad85cb, NAME => 'testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 992f98999ce7b2c13e6adb6eca28ff70, NAME => 'testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-13 21:57:51,782 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325471780"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325471780"}]},"ts":"1676325471780"} 2023-02-13 21:57:51,782 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325471780"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325471780"}]},"ts":"1676325471780"} 2023-02-13 21:57:51,782 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325471780"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325471780"}]},"ts":"1676325471780"} 2023-02-13 21:57:51,782 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325471780"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325471780"}]},"ts":"1676325471780"} 2023-02-13 21:57:51,782 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325471780"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325471780"}]},"ts":"1676325471780"} 2023-02-13 21:57:51,782 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325471780"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325471780"}]},"ts":"1676325471780"} 2023-02-13 21:57:51,784 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-13 21:57:51,784 INFO [PEWorker-1] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-13 21:57:51,784 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 1f6dc79bf11da8444a0870d779d2fb4d, NAME => 'testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 251bc68b009792a14e42934b238a6375, NAME => 'testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 44e821dce55cb29b05989bca574cff62, NAME => 'testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 4e757d660af6a5aaa6e2c43f8b91311d, NAME => 'testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 676fb98d6cc62f0fdbda4ce59aad85cb, NAME => 'testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 992f98999ce7b2c13e6adb6eca28ff70, NAME => 'testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-13 21:57:51,784 INFO [PEWorker-1] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb1-1676325454277 table=testRestoreSnapshot-1676325454277 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676325454277 completed! 2023-02-13 21:57:51,787 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=147, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676325454277 snapshot=name: "snaptb1-1676325454277" table: "testRestoreSnapshot-1676325454277" creation_time: 1676325465457 type: DISABLED version: 2 ttl: 0 ) in 139 msec 2023-02-13 21:57:51,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-02-13 21:57:51,902 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676325454277, procId: 147 completed 2023-02-13 21:57:51,902 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb1-1676325454277-1676325470866 2023-02-13 21:57:51,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(748): Client=jenkins//148.251.75.209 delete name: "hbase-failsafe-snaptb1-1676325454277-1676325470866" 2023-02-13 21:57:51,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb1-1676325454277-1676325470866 2023-02-13 21:57:51,906 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676325454277 2023-02-13 21:57:51,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.HMaster$10(2372): Client=jenkins//148.251.75.209 enable testRestoreSnapshot-1676325454277 2023-02-13 21:57:51,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=148, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:51,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-13 21:57:51,912 INFO [PEWorker-4] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676325454277 2023-02-13 21:57:51,913 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325471912"}]},"ts":"1676325471912"} 2023-02-13 21:57:51,914 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676325454277, state=ENABLING in hbase:meta 2023-02-13 21:57:51,916 DEBUG [PEWorker-4] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 21:57:51,916 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, ASSIGN}, {pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, ASSIGN}, {pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, ASSIGN}, {pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, ASSIGN}, {pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, ASSIGN}, {pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, ASSIGN}] 2023-02-13 21:57:51,918 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, ASSIGN 2023-02-13 21:57:51,918 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, ASSIGN 2023-02-13 21:57:51,918 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, ASSIGN 2023-02-13 21:57:51,918 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, ASSIGN 2023-02-13 21:57:51,919 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 21:57:51,919 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 21:57:51,919 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 21:57:51,919 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, ASSIGN 2023-02-13 21:57:51,919 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 21:57:51,919 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, ASSIGN 2023-02-13 21:57:51,919 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 21:57:51,920 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 21:57:52,069 DEBUG [jenkins-hbase20:46557] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-02-13 21:57:52,069 DEBUG [jenkins-hbase20:46557] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 21:57:52,069 DEBUG [jenkins-hbase20:46557] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 21:57:52,069 DEBUG [jenkins-hbase20:46557] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 21:57:52,069 DEBUG [jenkins-hbase20:46557] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 21:57:52,072 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=1f6dc79bf11da8444a0870d779d2fb4d, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:52,072 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=251bc68b009792a14e42934b238a6375, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:52,072 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=992f98999ce7b2c13e6adb6eca28ff70, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:52,073 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325472072"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325472072"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325472072"}]},"ts":"1676325472072"} 2023-02-13 21:57:52,073 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325472072"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325472072"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325472072"}]},"ts":"1676325472072"} 2023-02-13 21:57:52,072 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=676fb98d6cc62f0fdbda4ce59aad85cb, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:52,072 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=4e757d660af6a5aaa6e2c43f8b91311d, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:52,073 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325472072"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325472072"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325472072"}]},"ts":"1676325472072"} 2023-02-13 21:57:52,073 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325472072"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325472072"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325472072"}]},"ts":"1676325472072"} 2023-02-13 21:57:52,073 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325472072"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325472072"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325472072"}]},"ts":"1676325472072"} 2023-02-13 21:57:52,074 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=155, ppid=154, state=RUNNABLE; OpenRegionProcedure 251bc68b009792a14e42934b238a6375, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:57:52,075 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=156, ppid=150, state=RUNNABLE; OpenRegionProcedure 992f98999ce7b2c13e6adb6eca28ff70, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:57:52,075 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=157, ppid=152, state=RUNNABLE; OpenRegionProcedure 676fb98d6cc62f0fdbda4ce59aad85cb, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:57:52,076 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=158, ppid=153, state=RUNNABLE; OpenRegionProcedure 4e757d660af6a5aaa6e2c43f8b91311d, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:57:52,077 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=159, ppid=149, state=RUNNABLE; OpenRegionProcedure 1f6dc79bf11da8444a0870d779d2fb4d, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:57:52,079 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=44e821dce55cb29b05989bca574cff62, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:52,079 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325472079"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325472079"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325472079"}]},"ts":"1676325472079"} 2023-02-13 21:57:52,081 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=160, ppid=151, state=RUNNABLE; OpenRegionProcedure 44e821dce55cb29b05989bca574cff62, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:57:52,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-13 21:57:52,228 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:52,229 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 251bc68b009792a14e42934b238a6375, NAME => 'testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 21:57:52,229 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:52,229 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:52,229 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:52,229 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:52,231 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:52,231 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 992f98999ce7b2c13e6adb6eca28ff70, NAME => 'testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 21:57:52,231 INFO [StoreOpener-251bc68b009792a14e42934b238a6375-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:52,231 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:52,231 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:52,231 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:52,231 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:52,231 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:52,231 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 676fb98d6cc62f0fdbda4ce59aad85cb, NAME => 'testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 21:57:52,232 DEBUG [StoreOpener-251bc68b009792a14e42934b238a6375-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf 2023-02-13 21:57:52,232 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:52,232 DEBUG [StoreOpener-251bc68b009792a14e42934b238a6375-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf 2023-02-13 21:57:52,232 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:52,232 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:52,232 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:52,232 INFO [StoreOpener-251bc68b009792a14e42934b238a6375-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 251bc68b009792a14e42934b238a6375 columnFamilyName cf 2023-02-13 21:57:52,232 INFO [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:52,233 INFO [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:52,233 DEBUG [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf 2023-02-13 21:57:52,233 DEBUG [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf 2023-02-13 21:57:52,233 DEBUG [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf 2023-02-13 21:57:52,233 DEBUG [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf 2023-02-13 21:57:52,233 INFO [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 992f98999ce7b2c13e6adb6eca28ff70 columnFamilyName cf 2023-02-13 21:57:52,234 INFO [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 676fb98d6cc62f0fdbda4ce59aad85cb columnFamilyName cf 2023-02-13 21:57:52,250 DEBUG [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/testRestoreSnapshot-1676325454277=992f98999ce7b2c13e6adb6eca28ff70-46bad1cc5702482388220a80e425cbe0 2023-02-13 21:57:52,250 DEBUG [StoreOpener-251bc68b009792a14e42934b238a6375-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/testRestoreSnapshot-1676325454277=251bc68b009792a14e42934b238a6375-136600ed36234d64b213b50e0d6f02a9 2023-02-13 21:57:52,250 DEBUG [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/testRestoreSnapshot-1676325454277=676fb98d6cc62f0fdbda4ce59aad85cb-24e696c302c64bd2bfbbbddca37e4305 2023-02-13 21:57:52,258 DEBUG [StoreOpener-251bc68b009792a14e42934b238a6375-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/testRestoreSnapshot-1676325454277=251bc68b009792a14e42934b238a6375-29f391c114814c4b96610425ae0e5de0 2023-02-13 21:57:52,258 INFO [StoreOpener-251bc68b009792a14e42934b238a6375-1] regionserver.HStore(310): Store=251bc68b009792a14e42934b238a6375/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:52,259 DEBUG [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/testRestoreSnapshot-1676325454277=676fb98d6cc62f0fdbda4ce59aad85cb-99375a84534848d588fa45ac3699c454 2023-02-13 21:57:52,259 INFO [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] regionserver.HStore(310): Store=676fb98d6cc62f0fdbda4ce59aad85cb/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:52,260 DEBUG [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/testRestoreSnapshot-1676325454277=992f98999ce7b2c13e6adb6eca28ff70-ebd1b06b1c10407b903431c38f993f0f 2023-02-13 21:57:52,260 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:52,260 INFO [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] regionserver.HStore(310): Store=992f98999ce7b2c13e6adb6eca28ff70/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:52,261 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:52,261 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:52,262 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:52,264 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:52,264 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 676fb98d6cc62f0fdbda4ce59aad85cb; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64927295, jitterRate=-0.03250791132450104}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:52,264 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 676fb98d6cc62f0fdbda4ce59aad85cb: Running coprocessor pre-open hook at 1676325472232Writing region info on filesystem at 1676325472232Initializing all the Stores at 1676325472232Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325472233 (+1 ms)Cleaning up temporary data from old regions at 1676325472261 (+28 ms)Cleaning up detritus from prior splits at 1676325472262 (+1 ms)Running coprocessor post-open hooks at 1676325472264 (+2 ms)Region opened successfully at 1676325472264 2023-02-13 21:57:52,265 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:52,265 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb., pid=157, masterSystemTime=1676325472229 2023-02-13 21:57:52,265 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375 2023-02-13 21:57:52,266 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 992f98999ce7b2c13e6adb6eca28ff70; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61482160, jitterRate=-0.08384442329406738}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:52,266 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375 2023-02-13 21:57:52,266 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 992f98999ce7b2c13e6adb6eca28ff70: Running coprocessor pre-open hook at 1676325472231Writing region info on filesystem at 1676325472231Initializing all the Stores at 1676325472232 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325472232Cleaning up temporary data from old regions at 1676325472262 (+30 ms)Cleaning up detritus from prior splits at 1676325472263 (+1 ms)Running coprocessor post-open hooks at 1676325472266 (+3 ms)Region opened successfully at 1676325472266 2023-02-13 21:57:52,267 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70., pid=156, masterSystemTime=1676325472228 2023-02-13 21:57:52,268 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:52,269 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=676fb98d6cc62f0fdbda4ce59aad85cb, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:52,269 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:52,269 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:52,269 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:52,269 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=992f98999ce7b2c13e6adb6eca28ff70, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:52,269 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:52,270 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:52,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1f6dc79bf11da8444a0870d779d2fb4d, NAME => 'testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 21:57:52,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:52,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:52,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:52,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:52,269 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 4e757d660af6a5aaa6e2c43f8b91311d, NAME => 'testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 21:57:52,269 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325472269"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325472269"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325472269"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325472269"}]},"ts":"1676325472269"} 2023-02-13 21:57:52,270 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325472269"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325472269"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325472269"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325472269"}]},"ts":"1676325472269"} 2023-02-13 21:57:52,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:52,271 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:52,271 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:52,271 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:52,272 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:52,272 INFO [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:52,273 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 251bc68b009792a14e42934b238a6375; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61685993, jitterRate=-0.08080707490444183}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:52,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 251bc68b009792a14e42934b238a6375: Running coprocessor pre-open hook at 1676325472230Writing region info on filesystem at 1676325472230Initializing all the Stores at 1676325472230Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325472230Cleaning up temporary data from old regions at 1676325472266 (+36 ms)Cleaning up detritus from prior splits at 1676325472269 (+3 ms)Running coprocessor post-open hooks at 1676325472273 (+4 ms)Region opened successfully at 1676325472273 2023-02-13 21:57:52,274 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375., pid=155, masterSystemTime=1676325472226 2023-02-13 21:57:52,275 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=157, resume processing ppid=152 2023-02-13 21:57:52,275 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=157, ppid=152, state=SUCCESS; OpenRegionProcedure 676fb98d6cc62f0fdbda4ce59aad85cb, server=jenkins-hbase20.apache.org,39523,1676325451480 in 197 msec 2023-02-13 21:57:52,275 DEBUG [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf 2023-02-13 21:57:52,276 DEBUG [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf 2023-02-13 21:57:52,276 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=156, resume processing ppid=150 2023-02-13 21:57:52,276 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=156, ppid=150, state=SUCCESS; OpenRegionProcedure 992f98999ce7b2c13e6adb6eca28ff70, server=jenkins-hbase20.apache.org,45015,1676325451448 in 198 msec 2023-02-13 21:57:52,276 INFO [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1f6dc79bf11da8444a0870d779d2fb4d columnFamilyName cf 2023-02-13 21:57:52,276 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:52,277 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:52,277 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:52,277 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 44e821dce55cb29b05989bca574cff62, NAME => 'testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 21:57:52,277 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:52,277 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=152, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, ASSIGN in 359 msec 2023-02-13 21:57:52,277 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=251bc68b009792a14e42934b238a6375, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:52,277 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325472277"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325472277"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325472277"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325472277"}]},"ts":"1676325472277"} 2023-02-13 21:57:52,277 INFO [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:52,277 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:52,278 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:52,278 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:52,279 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=150, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, ASSIGN in 360 msec 2023-02-13 21:57:52,282 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=155, resume processing ppid=154 2023-02-13 21:57:52,282 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=155, ppid=154, state=SUCCESS; OpenRegionProcedure 251bc68b009792a14e42934b238a6375, server=jenkins-hbase20.apache.org,36339,1676325451407 in 206 msec 2023-02-13 21:57:52,283 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=154, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, ASSIGN in 366 msec 2023-02-13 21:57:52,287 DEBUG [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf 2023-02-13 21:57:52,287 DEBUG [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf 2023-02-13 21:57:52,287 INFO [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4e757d660af6a5aaa6e2c43f8b91311d columnFamilyName cf 2023-02-13 21:57:52,290 INFO [StoreOpener-44e821dce55cb29b05989bca574cff62-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:52,292 DEBUG [StoreOpener-44e821dce55cb29b05989bca574cff62-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf 2023-02-13 21:57:52,292 DEBUG [StoreOpener-44e821dce55cb29b05989bca574cff62-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf 2023-02-13 21:57:52,293 INFO [StoreOpener-44e821dce55cb29b05989bca574cff62-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 44e821dce55cb29b05989bca574cff62 columnFamilyName cf 2023-02-13 21:57:52,307 DEBUG [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/testRestoreSnapshot-1676325454277=1f6dc79bf11da8444a0870d779d2fb4d-3d85d09beeee48a5b67b39f0abb4165e 2023-02-13 21:57:52,313 DEBUG [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/testRestoreSnapshot-1676325454277=1f6dc79bf11da8444a0870d779d2fb4d-ecd3475741414bf0a2161d2b27bc209a 2023-02-13 21:57:52,314 INFO [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] regionserver.HStore(310): Store=1f6dc79bf11da8444a0870d779d2fb4d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:52,314 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:52,321 DEBUG [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/testRestoreSnapshot-1676325454277=4e757d660af6a5aaa6e2c43f8b91311d-08e8b394e04742929b62615c32c8ed4c 2023-02-13 21:57:52,322 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:52,323 DEBUG [StoreOpener-44e821dce55cb29b05989bca574cff62-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/testRestoreSnapshot-1676325454277=44e821dce55cb29b05989bca574cff62-8795edd282b341fba51b1c197d77b4f9 2023-02-13 21:57:52,327 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:52,330 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 1f6dc79bf11da8444a0870d779d2fb4d; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69840703, jitterRate=0.04070757329463959}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:52,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 1f6dc79bf11da8444a0870d779d2fb4d: Running coprocessor pre-open hook at 1676325472270Writing region info on filesystem at 1676325472270Initializing all the Stores at 1676325472272 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325472272Cleaning up temporary data from old regions at 1676325472322 (+50 ms)Cleaning up detritus from prior splits at 1676325472323 (+1 ms)Running coprocessor post-open hooks at 1676325472330 (+7 ms)Region opened successfully at 1676325472330 2023-02-13 21:57:52,331 DEBUG [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/testRestoreSnapshot-1676325454277=4e757d660af6a5aaa6e2c43f8b91311d-81bb121c83f84b0997b6675992ad85a1 2023-02-13 21:57:52,331 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d., pid=159, masterSystemTime=1676325472229 2023-02-13 21:57:52,331 INFO [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] regionserver.HStore(310): Store=4e757d660af6a5aaa6e2c43f8b91311d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:52,332 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:52,333 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:52,333 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:52,334 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=1f6dc79bf11da8444a0870d779d2fb4d, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:52,334 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325472333"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325472333"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325472333"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325472333"}]},"ts":"1676325472333"} 2023-02-13 21:57:52,334 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:52,340 DEBUG [StoreOpener-44e821dce55cb29b05989bca574cff62-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/testRestoreSnapshot-1676325454277=44e821dce55cb29b05989bca574cff62-f9ada1ebb3d0446a9743d17910965899 2023-02-13 21:57:52,340 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:52,340 INFO [StoreOpener-44e821dce55cb29b05989bca574cff62-1] regionserver.HStore(310): Store=44e821dce55cb29b05989bca574cff62/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:52,341 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=159, resume processing ppid=149 2023-02-13 21:57:52,341 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=159, ppid=149, state=SUCCESS; OpenRegionProcedure 1f6dc79bf11da8444a0870d779d2fb4d, server=jenkins-hbase20.apache.org,39523,1676325451480 in 259 msec 2023-02-13 21:57:52,341 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:52,341 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 4e757d660af6a5aaa6e2c43f8b91311d; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59400663, jitterRate=-0.1148611456155777}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:52,341 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 4e757d660af6a5aaa6e2c43f8b91311d: Running coprocessor pre-open hook at 1676325472271Writing region info on filesystem at 1676325472271Initializing all the Stores at 1676325472272 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325472272Cleaning up temporary data from old regions at 1676325472335 (+63 ms)Cleaning up detritus from prior splits at 1676325472336 (+1 ms)Running coprocessor post-open hooks at 1676325472341 (+5 ms)Region opened successfully at 1676325472341 2023-02-13 21:57:52,342 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d., pid=158, masterSystemTime=1676325472228 2023-02-13 21:57:52,343 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:52,344 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=149, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, ASSIGN in 425 msec 2023-02-13 21:57:52,344 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:52,344 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:52,345 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=4e757d660af6a5aaa6e2c43f8b91311d, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:52,345 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325472345"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325472345"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325472345"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325472345"}]},"ts":"1676325472345"} 2023-02-13 21:57:52,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:52,350 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=158, resume processing ppid=153 2023-02-13 21:57:52,350 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=158, ppid=153, state=SUCCESS; OpenRegionProcedure 4e757d660af6a5aaa6e2c43f8b91311d, server=jenkins-hbase20.apache.org,45015,1676325451448 in 271 msec 2023-02-13 21:57:52,350 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 44e821dce55cb29b05989bca574cff62; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62178333, jitterRate=-0.07347063720226288}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:52,350 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 44e821dce55cb29b05989bca574cff62: Running coprocessor pre-open hook at 1676325472278Writing region info on filesystem at 1676325472278Initializing all the Stores at 1676325472279 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325472279Cleaning up temporary data from old regions at 1676325472343 (+64 ms)Cleaning up detritus from prior splits at 1676325472344 (+1 ms)Running coprocessor post-open hooks at 1676325472350 (+6 ms)Region opened successfully at 1676325472350 2023-02-13 21:57:52,351 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62., pid=160, masterSystemTime=1676325472226 2023-02-13 21:57:52,352 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=153, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, ASSIGN in 434 msec 2023-02-13 21:57:52,353 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:52,353 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:52,354 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=44e821dce55cb29b05989bca574cff62, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:52,354 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325472354"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325472354"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325472354"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325472354"}]},"ts":"1676325472354"} 2023-02-13 21:57:52,359 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=160, resume processing ppid=151 2023-02-13 21:57:52,359 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=160, ppid=151, state=SUCCESS; OpenRegionProcedure 44e821dce55cb29b05989bca574cff62, server=jenkins-hbase20.apache.org,36339,1676325451407 in 275 msec 2023-02-13 21:57:52,361 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=151, resume processing ppid=148 2023-02-13 21:57:52,361 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=151, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, ASSIGN in 443 msec 2023-02-13 21:57:52,362 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325472362"}]},"ts":"1676325472362"} 2023-02-13 21:57:52,364 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676325454277, state=ENABLED in hbase:meta 2023-02-13 21:57:52,365 INFO [PEWorker-5] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676325454277' was successfully enabled. 2023-02-13 21:57:52,367 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=148, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676325454277 in 459 msec 2023-02-13 21:57:52,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-13 21:57:52,663 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676325454277, procId: 148 completed 2023-02-13 21:57:52,664 DEBUG [Listener at localhost.localdomain/44075] zookeeper.ReadOnlyZKClient(139): Connect 0x27ddad26 to 127.0.0.1:58580 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 21:57:52,673 DEBUG [Listener at localhost.localdomain/44075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@452ab791, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 21:57:52,674 DEBUG [hconnection-0x28933828-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 21:57:52,678 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:53102, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 21:57:52,682 DEBUG [hconnection-0x28933828-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 21:57:52,683 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44938, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 21:57:52,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,930 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 21:57:52,934 DEBUG [hconnection-0x28933828-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 21:57:52,935 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49716, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 21:57:52,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:52,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,255 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 21:57:53,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,560 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 21:57:53,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,886 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 21:57:53,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:53,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,243 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 21:57:54,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,437 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x356c47e3 to 127.0.0.1:58580 2023-02-13 21:57:54,437 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 21:57:54,437 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x555200b1 to 127.0.0.1:58580 2023-02-13 21:57:54,437 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 21:57:54,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:54,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,643 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 21:57:55,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:55,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:56,550 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676325454277 2023-02-13 21:57:56,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.HMaster$11(2405): Client=jenkins//148.251.75.209 disable testRestoreSnapshot-1676325454277 2023-02-13 21:57:56,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=161, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:56,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-13 21:57:56,554 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325476554"}]},"ts":"1676325476554"} 2023-02-13 21:57:56,555 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676325454277, state=DISABLING in hbase:meta 2023-02-13 21:57:56,557 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676325454277 to state=DISABLING 2023-02-13 21:57:56,557 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, UNASSIGN}, {pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, UNASSIGN}, {pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, UNASSIGN}, {pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, UNASSIGN}, {pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, UNASSIGN}, {pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, UNASSIGN}] 2023-02-13 21:57:56,559 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, UNASSIGN 2023-02-13 21:57:56,559 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, UNASSIGN 2023-02-13 21:57:56,559 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, UNASSIGN 2023-02-13 21:57:56,559 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, UNASSIGN 2023-02-13 21:57:56,559 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, UNASSIGN 2023-02-13 21:57:56,560 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=4e757d660af6a5aaa6e2c43f8b91311d, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:56,560 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=251bc68b009792a14e42934b238a6375, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:56,560 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325476560"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325476560"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325476560"}]},"ts":"1676325476560"} 2023-02-13 21:57:56,560 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325476560"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325476560"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325476560"}]},"ts":"1676325476560"} 2023-02-13 21:57:56,560 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=992f98999ce7b2c13e6adb6eca28ff70, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:56,560 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325476560"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325476560"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325476560"}]},"ts":"1676325476560"} 2023-02-13 21:57:56,561 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=676fb98d6cc62f0fdbda4ce59aad85cb, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:56,561 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=1f6dc79bf11da8444a0870d779d2fb4d, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:56,561 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325476561"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325476561"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325476561"}]},"ts":"1676325476561"} 2023-02-13 21:57:56,561 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325476561"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325476561"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325476561"}]},"ts":"1676325476561"} 2023-02-13 21:57:56,561 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=168, ppid=167, state=RUNNABLE; CloseRegionProcedure 251bc68b009792a14e42934b238a6375, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:57:56,562 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=169, ppid=166, state=RUNNABLE; CloseRegionProcedure 4e757d660af6a5aaa6e2c43f8b91311d, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:57:56,563 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=170, ppid=163, state=RUNNABLE; CloseRegionProcedure 992f98999ce7b2c13e6adb6eca28ff70, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:57:56,564 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=171, ppid=165, state=RUNNABLE; CloseRegionProcedure 676fb98d6cc62f0fdbda4ce59aad85cb, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:57:56,565 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=172, ppid=162, state=RUNNABLE; CloseRegionProcedure 1f6dc79bf11da8444a0870d779d2fb4d, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:57:56,565 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, UNASSIGN 2023-02-13 21:57:56,566 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=44e821dce55cb29b05989bca574cff62, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:56,566 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325476566"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325476566"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325476566"}]},"ts":"1676325476566"} 2023-02-13 21:57:56,567 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=173, ppid=164, state=RUNNABLE; CloseRegionProcedure 44e821dce55cb29b05989bca574cff62, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:57:56,714 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:56,714 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:56,715 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 44e821dce55cb29b05989bca574cff62, disabling compactions & flushes 2023-02-13 21:57:56,715 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 992f98999ce7b2c13e6adb6eca28ff70, disabling compactions & flushes 2023-02-13 21:57:56,715 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:56,715 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:56,715 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:56,715 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:56,716 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. after waiting 0 ms 2023-02-13 21:57:56,716 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. after waiting 0 ms 2023-02-13 21:57:56,716 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:56,716 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:56,716 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:56,717 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 1f6dc79bf11da8444a0870d779d2fb4d, disabling compactions & flushes 2023-02-13 21:57:56,717 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:56,717 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:56,717 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. after waiting 0 ms 2023-02-13 21:57:56,717 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:56,726 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-13 21:57:56,726 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-13 21:57:56,726 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-13 21:57:56,727 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:56,728 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:56,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 1f6dc79bf11da8444a0870d779d2fb4d: Waiting for close lock at 1676325476716Running coprocessor pre-close hooks at 1676325476716Disabling compacts and flushes for region at 1676325476716Disabling writes for close at 1676325476717 (+1 ms)Writing region close event to WAL at 1676325476723 (+6 ms)Running coprocessor post-close hooks at 1676325476727 (+4 ms)Closed at 1676325476727 2023-02-13 21:57:56,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 992f98999ce7b2c13e6adb6eca28ff70: Waiting for close lock at 1676325476714Running coprocessor pre-close hooks at 1676325476714Disabling compacts and flushes for region at 1676325476714Disabling writes for close at 1676325476716 (+2 ms)Writing region close event to WAL at 1676325476722 (+6 ms)Running coprocessor post-close hooks at 1676325476727 (+5 ms)Closed at 1676325476728 (+1 ms) 2023-02-13 21:57:56,728 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:56,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 44e821dce55cb29b05989bca574cff62: Waiting for close lock at 1676325476714Running coprocessor pre-close hooks at 1676325476714Disabling compacts and flushes for region at 1676325476714Disabling writes for close at 1676325476716 (+2 ms)Writing region close event to WAL at 1676325476723 (+7 ms)Running coprocessor post-close hooks at 1676325476728 (+5 ms)Closed at 1676325476728 2023-02-13 21:57:56,730 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:56,730 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:56,731 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 4e757d660af6a5aaa6e2c43f8b91311d, disabling compactions & flushes 2023-02-13 21:57:56,731 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:56,731 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:56,731 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. after waiting 0 ms 2023-02-13 21:57:56,731 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:56,731 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=992f98999ce7b2c13e6adb6eca28ff70, regionState=CLOSED 2023-02-13 21:57:56,731 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325476731"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325476731"}]},"ts":"1676325476731"} 2023-02-13 21:57:56,731 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:56,731 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:56,732 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=1f6dc79bf11da8444a0870d779d2fb4d, regionState=CLOSED 2023-02-13 21:57:56,732 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325476732"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325476732"}]},"ts":"1676325476732"} 2023-02-13 21:57:56,732 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:56,732 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:56,733 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=44e821dce55cb29b05989bca574cff62, regionState=CLOSED 2023-02-13 21:57:56,733 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325476733"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325476733"}]},"ts":"1676325476733"} 2023-02-13 21:57:56,735 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 676fb98d6cc62f0fdbda4ce59aad85cb, disabling compactions & flushes 2023-02-13 21:57:56,735 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 251bc68b009792a14e42934b238a6375, disabling compactions & flushes 2023-02-13 21:57:56,736 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:56,736 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:56,736 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:56,736 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:56,736 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. after waiting 0 ms 2023-02-13 21:57:56,737 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:56,736 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. after waiting 0 ms 2023-02-13 21:57:56,739 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:56,744 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=170, resume processing ppid=163 2023-02-13 21:57:56,744 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=172, resume processing ppid=162 2023-02-13 21:57:56,744 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=173, resume processing ppid=164 2023-02-13 21:57:56,746 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=173, ppid=164, state=SUCCESS; CloseRegionProcedure 44e821dce55cb29b05989bca574cff62, server=jenkins-hbase20.apache.org,36339,1676325451407 in 167 msec 2023-02-13 21:57:56,746 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=172, ppid=162, state=SUCCESS; CloseRegionProcedure 1f6dc79bf11da8444a0870d779d2fb4d, server=jenkins-hbase20.apache.org,39523,1676325451480 in 169 msec 2023-02-13 21:57:56,746 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=170, ppid=163, state=SUCCESS; CloseRegionProcedure 992f98999ce7b2c13e6adb6eca28ff70, server=jenkins-hbase20.apache.org,45015,1676325451448 in 170 msec 2023-02-13 21:57:56,747 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-13 21:57:56,748 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=164, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, UNASSIGN in 187 msec 2023-02-13 21:57:56,748 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=162, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, UNASSIGN in 187 msec 2023-02-13 21:57:56,749 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=163, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, UNASSIGN in 189 msec 2023-02-13 21:57:56,749 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:56,749 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 4e757d660af6a5aaa6e2c43f8b91311d: Waiting for close lock at 1676325476730Running coprocessor pre-close hooks at 1676325476730Disabling compacts and flushes for region at 1676325476730Disabling writes for close at 1676325476731 (+1 ms)Writing region close event to WAL at 1676325476739 (+8 ms)Running coprocessor post-close hooks at 1676325476749 (+10 ms)Closed at 1676325476749 2023-02-13 21:57:56,750 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-13 21:57:56,751 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:56,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 251bc68b009792a14e42934b238a6375: Waiting for close lock at 1676325476732Running coprocessor pre-close hooks at 1676325476732Disabling compacts and flushes for region at 1676325476732Disabling writes for close at 1676325476737 (+5 ms)Writing region close event to WAL at 1676325476747 (+10 ms)Running coprocessor post-close hooks at 1676325476751 (+4 ms)Closed at 1676325476751 2023-02-13 21:57:56,752 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:56,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-13 21:57:56,752 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=4e757d660af6a5aaa6e2c43f8b91311d, regionState=CLOSED 2023-02-13 21:57:56,752 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325476752"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325476752"}]},"ts":"1676325476752"} 2023-02-13 21:57:56,753 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:56,754 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 676fb98d6cc62f0fdbda4ce59aad85cb: Waiting for close lock at 1676325476731Running coprocessor pre-close hooks at 1676325476731Disabling compacts and flushes for region at 1676325476731Disabling writes for close at 1676325476739 (+8 ms)Writing region close event to WAL at 1676325476748 (+9 ms)Running coprocessor post-close hooks at 1676325476753 (+5 ms)Closed at 1676325476753 2023-02-13 21:57:56,754 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:56,754 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=251bc68b009792a14e42934b238a6375, regionState=CLOSED 2023-02-13 21:57:56,754 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325476754"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325476754"}]},"ts":"1676325476754"} 2023-02-13 21:57:56,755 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:56,755 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=676fb98d6cc62f0fdbda4ce59aad85cb, regionState=CLOSED 2023-02-13 21:57:56,756 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325476755"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325476755"}]},"ts":"1676325476755"} 2023-02-13 21:57:56,756 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=169, resume processing ppid=166 2023-02-13 21:57:56,756 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=169, ppid=166, state=SUCCESS; CloseRegionProcedure 4e757d660af6a5aaa6e2c43f8b91311d, server=jenkins-hbase20.apache.org,45015,1676325451448 in 192 msec 2023-02-13 21:57:56,757 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=168, resume processing ppid=167 2023-02-13 21:57:56,757 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=168, ppid=167, state=SUCCESS; CloseRegionProcedure 251bc68b009792a14e42934b238a6375, server=jenkins-hbase20.apache.org,36339,1676325451407 in 194 msec 2023-02-13 21:57:56,757 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=166, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, UNASSIGN in 199 msec 2023-02-13 21:57:56,758 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=171, resume processing ppid=165 2023-02-13 21:57:56,758 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=171, ppid=165, state=SUCCESS; CloseRegionProcedure 676fb98d6cc62f0fdbda4ce59aad85cb, server=jenkins-hbase20.apache.org,39523,1676325451480 in 193 msec 2023-02-13 21:57:56,758 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=167, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, UNASSIGN in 200 msec 2023-02-13 21:57:56,761 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=165, resume processing ppid=161 2023-02-13 21:57:56,761 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=165, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, UNASSIGN in 201 msec 2023-02-13 21:57:56,761 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325476761"}]},"ts":"1676325476761"} 2023-02-13 21:57:56,762 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676325454277, state=DISABLED in hbase:meta 2023-02-13 21:57:56,763 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676325454277 to state=DISABLED 2023-02-13 21:57:56,765 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=161, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676325454277 in 213 msec 2023-02-13 21:57:56,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-13 21:57:56,806 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676325454277, procId: 161 completed 2023-02-13 21:57:56,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.HMaster$5(2215): Client=jenkins//148.251.75.209 delete testRestoreSnapshot-1676325454277 2023-02-13 21:57:56,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:56,819 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:56,820 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=174, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:56,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-02-13 21:57:56,827 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:56,827 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375 2023-02-13 21:57:56,827 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:56,827 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:56,827 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:56,827 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:56,829 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf, FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/recovered.edits] 2023-02-13 21:57:56,829 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf, FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/recovered.edits] 2023-02-13 21:57:56,829 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf, FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/recovered.edits] 2023-02-13 21:57:56,829 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf, FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/recovered.edits] 2023-02-13 21:57:56,830 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf, FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/recovered.edits] 2023-02-13 21:57:56,830 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf, FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/recovered.edits] 2023-02-13 21:57:56,832 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/testRestoreSnapshot-1676325454277=676fb98d6cc62f0fdbda4ce59aad85cb-24e696c302c64bd2bfbbbddca37e4305 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/testRestoreSnapshot-1676325454277=676fb98d6cc62f0fdbda4ce59aad85cb-24e696c302c64bd2bfbbbddca37e4305 2023-02-13 21:57:56,833 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/testRestoreSnapshot-1676325454277=251bc68b009792a14e42934b238a6375-136600ed36234d64b213b50e0d6f02a9 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/testRestoreSnapshot-1676325454277=251bc68b009792a14e42934b238a6375-136600ed36234d64b213b50e0d6f02a9 2023-02-13 21:57:56,833 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/testRestoreSnapshot-1676325454277=4e757d660af6a5aaa6e2c43f8b91311d-08e8b394e04742929b62615c32c8ed4c to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/testRestoreSnapshot-1676325454277=4e757d660af6a5aaa6e2c43f8b91311d-08e8b394e04742929b62615c32c8ed4c 2023-02-13 21:57:56,833 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/testRestoreSnapshot-1676325454277=992f98999ce7b2c13e6adb6eca28ff70-46bad1cc5702482388220a80e425cbe0 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/testRestoreSnapshot-1676325454277=992f98999ce7b2c13e6adb6eca28ff70-46bad1cc5702482388220a80e425cbe0 2023-02-13 21:57:56,833 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/testRestoreSnapshot-1676325454277=44e821dce55cb29b05989bca574cff62-8795edd282b341fba51b1c197d77b4f9 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/testRestoreSnapshot-1676325454277=44e821dce55cb29b05989bca574cff62-8795edd282b341fba51b1c197d77b4f9 2023-02-13 21:57:56,834 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/testRestoreSnapshot-1676325454277=1f6dc79bf11da8444a0870d779d2fb4d-3d85d09beeee48a5b67b39f0abb4165e to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/testRestoreSnapshot-1676325454277=1f6dc79bf11da8444a0870d779d2fb4d-3d85d09beeee48a5b67b39f0abb4165e 2023-02-13 21:57:56,834 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/testRestoreSnapshot-1676325454277=676fb98d6cc62f0fdbda4ce59aad85cb-99375a84534848d588fa45ac3699c454 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/testRestoreSnapshot-1676325454277=676fb98d6cc62f0fdbda4ce59aad85cb-99375a84534848d588fa45ac3699c454 2023-02-13 21:57:56,835 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/testRestoreSnapshot-1676325454277=251bc68b009792a14e42934b238a6375-29f391c114814c4b96610425ae0e5de0 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/testRestoreSnapshot-1676325454277=251bc68b009792a14e42934b238a6375-29f391c114814c4b96610425ae0e5de0 2023-02-13 21:57:56,835 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/testRestoreSnapshot-1676325454277=4e757d660af6a5aaa6e2c43f8b91311d-81bb121c83f84b0997b6675992ad85a1 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/testRestoreSnapshot-1676325454277=4e757d660af6a5aaa6e2c43f8b91311d-81bb121c83f84b0997b6675992ad85a1 2023-02-13 21:57:56,836 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/testRestoreSnapshot-1676325454277=992f98999ce7b2c13e6adb6eca28ff70-ebd1b06b1c10407b903431c38f993f0f to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/testRestoreSnapshot-1676325454277=992f98999ce7b2c13e6adb6eca28ff70-ebd1b06b1c10407b903431c38f993f0f 2023-02-13 21:57:56,836 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/testRestoreSnapshot-1676325454277=44e821dce55cb29b05989bca574cff62-f9ada1ebb3d0446a9743d17910965899 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/testRestoreSnapshot-1676325454277=44e821dce55cb29b05989bca574cff62-f9ada1ebb3d0446a9743d17910965899 2023-02-13 21:57:56,836 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/testRestoreSnapshot-1676325454277=1f6dc79bf11da8444a0870d779d2fb4d-ecd3475741414bf0a2161d2b27bc209a to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/testRestoreSnapshot-1676325454277=1f6dc79bf11da8444a0870d779d2fb4d-ecd3475741414bf0a2161d2b27bc209a 2023-02-13 21:57:56,841 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/recovered.edits/27.seqid to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/recovered.edits/27.seqid 2023-02-13 21:57:56,841 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:56,847 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/recovered.edits/27.seqid to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/recovered.edits/27.seqid 2023-02-13 21:57:56,848 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:56,848 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/recovered.edits/27.seqid to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/recovered.edits/27.seqid 2023-02-13 21:57:56,848 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/recovered.edits/27.seqid to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/recovered.edits/27.seqid 2023-02-13 21:57:56,848 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/recovered.edits/27.seqid to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/recovered.edits/27.seqid 2023-02-13 21:57:56,848 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:56,848 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:56,849 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:56,851 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/recovered.edits/27.seqid to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/recovered.edits/27.seqid 2023-02-13 21:57:56,851 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375 2023-02-13 21:57:56,851 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676325454277 regions 2023-02-13 21:57:56,852 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881 2023-02-13 21:57:56,852 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf] 2023-02-13 21:57:56,855 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e 2023-02-13 21:57:56,856 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 2023-02-13 21:57:56,857 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 2023-02-13 21:57:56,858 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 2023-02-13 21:57:56,859 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 2023-02-13 21:57:56,860 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 2023-02-13 21:57:56,861 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e 2023-02-13 21:57:56,862 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 2023-02-13 21:57:56,863 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a 2023-02-13 21:57:56,864 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 2023-02-13 21:57:56,865 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 2023-02-13 21:57:56,865 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b 2023-02-13 21:57:56,866 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881 2023-02-13 21:57:56,868 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=174, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:56,891 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1676325454277 from hbase:meta 2023-02-13 21:57:56,893 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1676325454277' descriptor. 2023-02-13 21:57:56,894 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=174, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:56,894 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1676325454277' from region states. 2023-02-13 21:57:56,894 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325476894"}]},"ts":"9223372036854775807"} 2023-02-13 21:57:56,894 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325476894"}]},"ts":"9223372036854775807"} 2023-02-13 21:57:56,895 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325476894"}]},"ts":"9223372036854775807"} 2023-02-13 21:57:56,895 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325476894"}]},"ts":"9223372036854775807"} 2023-02-13 21:57:56,895 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325476894"}]},"ts":"9223372036854775807"} 2023-02-13 21:57:56,895 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325476894"}]},"ts":"9223372036854775807"} 2023-02-13 21:57:56,897 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-13 21:57:56,897 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 1f6dc79bf11da8444a0870d779d2fb4d, NAME => 'testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 992f98999ce7b2c13e6adb6eca28ff70, NAME => 'testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 44e821dce55cb29b05989bca574cff62, NAME => 'testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 676fb98d6cc62f0fdbda4ce59aad85cb, NAME => 'testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 4e757d660af6a5aaa6e2c43f8b91311d, NAME => 'testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 251bc68b009792a14e42934b238a6375, NAME => 'testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.', STARTKEY => '5', ENDKEY => ''}] 2023-02-13 21:57:56,897 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1676325454277' as deleted. 2023-02-13 21:57:56,897 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676325476897"}]},"ts":"9223372036854775807"} 2023-02-13 21:57:56,898 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1676325454277 state from META 2023-02-13 21:57:56,900 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=174, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:56,901 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=174, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1676325454277 in 89 msec 2023-02-13 21:57:57,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-02-13 21:57:57,073 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1676325454277, procId: 174 completed 2023-02-13 21:57:57,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=175, state=RUNNABLE:CLONE_SNAPSHOT_PRE_OPERATION; CloneSnapshotProcedure (table=testRestoreSnapshot-1676325454277 snapshot=name: "snaptb1-1676325454277" table: "testRestoreSnapshot-1676325454277" creation_time: 1676325465457 type: DISABLED version: 2 ttl: 0 ) 2023-02-13 21:57:57,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(750): Clone snapshot=snaptb1-1676325454277 as table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:57,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-13 21:57:57,117 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/.tabledesc/.tableinfo.0000000001 2023-02-13 21:57:57,119 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1676325454277" table: "testRestoreSnapshot-1676325454277" creation_time: 1676325465457 type: DISABLED version: 2 ttl: 0 2023-02-13 21:57:57,119 DEBUG [PEWorker-4] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277 2023-02-13 21:57:57,119 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e from cloned region in snapshot snaptb1-1676325454277 to table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:57,122 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 from cloned region in snapshot snaptb1-1676325454277 to table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:57,124 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 from cloned region in snapshot snaptb1-1676325454277 to table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:57,127 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 from cloned region in snapshot snaptb1-1676325454277 to table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:57,129 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 from cloned region in snapshot snaptb1-1676325454277 to table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:57,131 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 from cloned region in snapshot snaptb1-1676325454277 to table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:57,133 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e from cloned region in snapshot snaptb1-1676325454277 to table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:57,135 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 from cloned region in snapshot snaptb1-1676325454277 to table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:57,137 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a from cloned region in snapshot snaptb1-1676325454277 to table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:57,138 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 from cloned region in snapshot snaptb1-1676325454277 to table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:57,140 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 from cloned region in snapshot snaptb1-1676325454277 to table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:57,142 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b from cloned region in snapshot snaptb1-1676325454277 to table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:57,144 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(239): region to add: 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:57,144 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(239): region to add: 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:57,144 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(239): region to add: 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:57,144 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(239): region to add: 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:57,144 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(239): region to add: 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:57,144 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(239): region to add: 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:57,144 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(585): clone region=676fb98d6cc62f0fdbda4ce59aad85cb as 676fb98d6cc62f0fdbda4ce59aad85cb in snapshot snaptb1-1676325454277 2023-02-13 21:57:57,144 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(585): clone region=251bc68b009792a14e42934b238a6375 as 251bc68b009792a14e42934b238a6375 in snapshot snaptb1-1676325454277 2023-02-13 21:57:57,144 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(585): clone region=44e821dce55cb29b05989bca574cff62 as 44e821dce55cb29b05989bca574cff62 in snapshot snaptb1-1676325454277 2023-02-13 21:57:57,145 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(585): clone region=992f98999ce7b2c13e6adb6eca28ff70 as 992f98999ce7b2c13e6adb6eca28ff70 in snapshot snaptb1-1676325454277 2023-02-13 21:57:57,145 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(585): clone region=1f6dc79bf11da8444a0870d779d2fb4d as 1f6dc79bf11da8444a0870d779d2fb4d in snapshot snaptb1-1676325454277 2023-02-13 21:57:57,145 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(585): clone region=4e757d660af6a5aaa6e2c43f8b91311d as 4e757d660af6a5aaa6e2c43f8b91311d in snapshot snaptb1-1676325454277 2023-02-13 21:57:57,146 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(7671): creating {ENCODED => 4e757d660af6a5aaa6e2c43f8b91311d, NAME => 'testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1676325454277', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp 2023-02-13 21:57:57,147 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(7671): creating {ENCODED => 992f98999ce7b2c13e6adb6eca28ff70, NAME => 'testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1676325454277', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp 2023-02-13 21:57:57,147 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(7671): creating {ENCODED => 44e821dce55cb29b05989bca574cff62, NAME => 'testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1676325454277', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp 2023-02-13 21:57:57,148 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(7671): creating {ENCODED => 251bc68b009792a14e42934b238a6375, NAME => 'testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1676325454277', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp 2023-02-13 21:57:57,148 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(7671): creating {ENCODED => 1f6dc79bf11da8444a0870d779d2fb4d, NAME => 'testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1676325454277', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp 2023-02-13 21:57:57,154 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(7671): creating {ENCODED => 676fb98d6cc62f0fdbda4ce59aad85cb, NAME => 'testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1676325454277', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp 2023-02-13 21:57:57,172 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:57,172 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 3d85d09beeee48a5b67b39f0abb4165e from cloned region in snapshot snaptb1-1676325454277 to table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:57,183 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:57,183 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 8795edd282b341fba51b1c197d77b4f9 from cloned region in snapshot snaptb1-1676325454277 to table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:57,190 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink ecd3475741414bf0a2161d2b27bc209a from cloned region in snapshot snaptb1-1676325454277 to table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:57,190 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:57,190 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 46bad1cc5702482388220a80e425cbe0 from cloned region in snapshot snaptb1-1676325454277 to table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:57,190 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:57,191 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 24e696c302c64bd2bfbbbddca37e4305 from cloned region in snapshot snaptb1-1676325454277 to table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:57,191 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink f9ada1ebb3d0446a9743d17910965899 from cloned region in snapshot snaptb1-1676325454277 to table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:57,192 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:57,193 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 136600ed36234d64b213b50e0d6f02a9 from cloned region in snapshot snaptb1-1676325454277 to table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:57,193 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:57,193 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 08e8b394e04742929b62615c32c8ed4c from cloned region in snapshot snaptb1-1676325454277 to table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:57,199 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink ebd1b06b1c10407b903431c38f993f0f from cloned region in snapshot snaptb1-1676325454277 to table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:57,200 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1603): Closing 1f6dc79bf11da8444a0870d779d2fb4d, disabling compactions & flushes 2023-02-13 21:57:57,200 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:57,200 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:57,200 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. after waiting 0 ms 2023-02-13 21:57:57,200 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:57,200 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:57,200 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1557): Region close journal for 1f6dc79bf11da8444a0870d779d2fb4d: Waiting for close lock at 1676325477200Disabling compacts and flushes for region at 1676325477200Disabling writes for close at 1676325477200Writing region close event to WAL at 1676325477200Closed at 1676325477200 2023-02-13 21:57:57,203 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 99375a84534848d588fa45ac3699c454 from cloned region in snapshot snaptb1-1676325454277 to table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:57,203 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 29f391c114814c4b96610425ae0e5de0 from cloned region in snapshot snaptb1-1676325454277 to table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:57,203 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1603): Closing 44e821dce55cb29b05989bca574cff62, disabling compactions & flushes 2023-02-13 21:57:57,203 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:57,203 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:57,203 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. after waiting 0 ms 2023-02-13 21:57:57,203 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:57,203 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:57,203 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1557): Region close journal for 44e821dce55cb29b05989bca574cff62: Waiting for close lock at 1676325477203Disabling compacts and flushes for region at 1676325477203Disabling writes for close at 1676325477203Writing region close event to WAL at 1676325477203Closed at 1676325477203 2023-02-13 21:57:57,204 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1603): Closing 992f98999ce7b2c13e6adb6eca28ff70, disabling compactions & flushes 2023-02-13 21:57:57,204 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 81bb121c83f84b0997b6675992ad85a1 from cloned region in snapshot snaptb1-1676325454277 to table=testRestoreSnapshot-1676325454277 2023-02-13 21:57:57,204 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:57,204 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:57,204 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. after waiting 0 ms 2023-02-13 21:57:57,204 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:57,204 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:57,204 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1557): Region close journal for 992f98999ce7b2c13e6adb6eca28ff70: Waiting for close lock at 1676325477204Disabling compacts and flushes for region at 1676325477204Disabling writes for close at 1676325477204Writing region close event to WAL at 1676325477204Closed at 1676325477204 2023-02-13 21:57:57,205 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1603): Closing 676fb98d6cc62f0fdbda4ce59aad85cb, disabling compactions & flushes 2023-02-13 21:57:57,205 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:57,205 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:57,205 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. after waiting 0 ms 2023-02-13 21:57:57,205 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:57,205 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:57,205 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1557): Region close journal for 676fb98d6cc62f0fdbda4ce59aad85cb: Waiting for close lock at 1676325477205Disabling compacts and flushes for region at 1676325477205Disabling writes for close at 1676325477205Writing region close event to WAL at 1676325477205Closed at 1676325477205 2023-02-13 21:57:57,205 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1603): Closing 251bc68b009792a14e42934b238a6375, disabling compactions & flushes 2023-02-13 21:57:57,205 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:57,205 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:57,206 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. after waiting 0 ms 2023-02-13 21:57:57,206 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:57,206 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:57,206 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1557): Region close journal for 251bc68b009792a14e42934b238a6375: Waiting for close lock at 1676325477205Disabling compacts and flushes for region at 1676325477205Disabling writes for close at 1676325477206 (+1 ms)Writing region close event to WAL at 1676325477206Closed at 1676325477206 2023-02-13 21:57:57,206 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1603): Closing 4e757d660af6a5aaa6e2c43f8b91311d, disabling compactions & flushes 2023-02-13 21:57:57,206 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:57,206 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:57,206 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. after waiting 0 ms 2023-02-13 21:57:57,206 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:57,206 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:57,206 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1557): Region close journal for 4e757d660af6a5aaa6e2c43f8b91311d: Waiting for close lock at 1676325477206Disabling compacts and flushes for region at 1676325477206Disabling writes for close at 1676325477206Writing region close event to WAL at 1676325477206Closed at 1676325477206 2023-02-13 21:57:57,206 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1676325454277" table: "testRestoreSnapshot-1676325454277" creation_time: 1676325465457 type: DISABLED version: 2 ttl: 0 2023-02-13 21:57:57,207 INFO [PEWorker-4] procedure.CloneSnapshotProcedure$1(411): Clone snapshot=snaptb1-1676325454277 on table=testRestoreSnapshot-1676325454277 completed! 2023-02-13 21:57:57,211 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325477211"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325477211"}]},"ts":"1676325477211"} 2023-02-13 21:57:57,211 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325477211"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325477211"}]},"ts":"1676325477211"} 2023-02-13 21:57:57,211 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325477211"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325477211"}]},"ts":"1676325477211"} 2023-02-13 21:57:57,211 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325477211"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325477211"}]},"ts":"1676325477211"} 2023-02-13 21:57:57,211 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325477211"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325477211"}]},"ts":"1676325477211"} 2023-02-13 21:57:57,211 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325477211"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325477211"}]},"ts":"1676325477211"} 2023-02-13 21:57:57,213 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-13 21:57:57,214 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325477214"}]},"ts":"1676325477214"} 2023-02-13 21:57:57,215 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676325454277, state=ENABLING in hbase:meta 2023-02-13 21:57:57,219 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-02-13 21:57:57,219 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 21:57:57,219 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 21:57:57,219 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 21:57:57,219 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 21:57:57,219 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, ASSIGN}, {pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, ASSIGN}, {pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, ASSIGN}, {pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, ASSIGN}, {pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, ASSIGN}, {pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, ASSIGN}] 2023-02-13 21:57:57,222 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, ASSIGN 2023-02-13 21:57:57,222 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, ASSIGN 2023-02-13 21:57:57,222 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, ASSIGN 2023-02-13 21:57:57,222 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, ASSIGN 2023-02-13 21:57:57,222 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, ASSIGN 2023-02-13 21:57:57,222 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,45015,1676325451448; forceNewPlan=false, retain=false 2023-02-13 21:57:57,222 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,36339,1676325451407; forceNewPlan=false, retain=false 2023-02-13 21:57:57,222 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,36339,1676325451407; forceNewPlan=false, retain=false 2023-02-13 21:57:57,222 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,39523,1676325451480; forceNewPlan=false, retain=false 2023-02-13 21:57:57,223 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,39523,1676325451480; forceNewPlan=false, retain=false 2023-02-13 21:57:57,223 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, ASSIGN 2023-02-13 21:57:57,224 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,45015,1676325451448; forceNewPlan=false, retain=false 2023-02-13 21:57:57,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-13 21:57:57,373 INFO [jenkins-hbase20:46557] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 21:57:57,376 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=992f98999ce7b2c13e6adb6eca28ff70, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:57,376 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=251bc68b009792a14e42934b238a6375, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:57,376 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=1f6dc79bf11da8444a0870d779d2fb4d, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:57,377 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325477376"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325477376"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325477376"}]},"ts":"1676325477376"} 2023-02-13 21:57:57,377 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325477376"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325477376"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325477376"}]},"ts":"1676325477376"} 2023-02-13 21:57:57,376 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=676fb98d6cc62f0fdbda4ce59aad85cb, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:57,376 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=4e757d660af6a5aaa6e2c43f8b91311d, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:57,377 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325477376"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325477376"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325477376"}]},"ts":"1676325477376"} 2023-02-13 21:57:57,377 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325477376"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325477376"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325477376"}]},"ts":"1676325477376"} 2023-02-13 21:57:57,376 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325477376"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325477376"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325477376"}]},"ts":"1676325477376"} 2023-02-13 21:57:57,379 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=182, ppid=181, state=RUNNABLE; OpenRegionProcedure 251bc68b009792a14e42934b238a6375, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:57:57,382 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=183, ppid=176, state=RUNNABLE; OpenRegionProcedure 1f6dc79bf11da8444a0870d779d2fb4d, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:57:57,383 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=184, ppid=179, state=RUNNABLE; OpenRegionProcedure 676fb98d6cc62f0fdbda4ce59aad85cb, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:57:57,384 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=185, ppid=180, state=RUNNABLE; OpenRegionProcedure 4e757d660af6a5aaa6e2c43f8b91311d, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:57:57,384 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=44e821dce55cb29b05989bca574cff62, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:57,384 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325477384"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325477384"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325477384"}]},"ts":"1676325477384"} 2023-02-13 21:57:57,385 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=186, ppid=177, state=RUNNABLE; OpenRegionProcedure 992f98999ce7b2c13e6adb6eca28ff70, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:57:57,387 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=187, ppid=178, state=RUNNABLE; OpenRegionProcedure 44e821dce55cb29b05989bca574cff62, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:57:57,535 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:57,535 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 251bc68b009792a14e42934b238a6375, NAME => 'testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 21:57:57,535 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:57,535 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:57,535 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:57,535 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:57,536 INFO [StoreOpener-251bc68b009792a14e42934b238a6375-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:57,536 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:57,537 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 992f98999ce7b2c13e6adb6eca28ff70, NAME => 'testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 21:57:57,537 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:57,537 DEBUG [StoreOpener-251bc68b009792a14e42934b238a6375-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf 2023-02-13 21:57:57,537 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:57,537 DEBUG [StoreOpener-251bc68b009792a14e42934b238a6375-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf 2023-02-13 21:57:57,537 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:57,537 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:57,537 INFO [StoreOpener-251bc68b009792a14e42934b238a6375-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 251bc68b009792a14e42934b238a6375 columnFamilyName cf 2023-02-13 21:57:57,538 INFO [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:57,538 DEBUG [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf 2023-02-13 21:57:57,538 DEBUG [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf 2023-02-13 21:57:57,539 INFO [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 992f98999ce7b2c13e6adb6eca28ff70 columnFamilyName cf 2023-02-13 21:57:57,539 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:57,539 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 4e757d660af6a5aaa6e2c43f8b91311d, NAME => 'testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 21:57:57,539 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:57,539 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:57,539 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:57,539 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:57,541 INFO [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:57,542 DEBUG [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf 2023-02-13 21:57:57,542 DEBUG [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf 2023-02-13 21:57:57,542 INFO [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4e757d660af6a5aaa6e2c43f8b91311d columnFamilyName cf 2023-02-13 21:57:57,549 DEBUG [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/testRestoreSnapshot-1676325454277=992f98999ce7b2c13e6adb6eca28ff70-46bad1cc5702482388220a80e425cbe0 2023-02-13 21:57:57,550 DEBUG [StoreOpener-251bc68b009792a14e42934b238a6375-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/testRestoreSnapshot-1676325454277=251bc68b009792a14e42934b238a6375-136600ed36234d64b213b50e0d6f02a9 2023-02-13 21:57:57,553 DEBUG [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/testRestoreSnapshot-1676325454277=4e757d660af6a5aaa6e2c43f8b91311d-08e8b394e04742929b62615c32c8ed4c 2023-02-13 21:57:57,558 DEBUG [StoreOpener-251bc68b009792a14e42934b238a6375-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/testRestoreSnapshot-1676325454277=251bc68b009792a14e42934b238a6375-29f391c114814c4b96610425ae0e5de0 2023-02-13 21:57:57,558 INFO [StoreOpener-251bc68b009792a14e42934b238a6375-1] regionserver.HStore(310): Store=251bc68b009792a14e42934b238a6375/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:57,560 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375 2023-02-13 21:57:57,560 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375 2023-02-13 21:57:57,561 DEBUG [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/testRestoreSnapshot-1676325454277=4e757d660af6a5aaa6e2c43f8b91311d-81bb121c83f84b0997b6675992ad85a1 2023-02-13 21:57:57,562 INFO [StoreOpener-4e757d660af6a5aaa6e2c43f8b91311d-1] regionserver.HStore(310): Store=4e757d660af6a5aaa6e2c43f8b91311d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:57,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:57,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:57,563 DEBUG [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/testRestoreSnapshot-1676325454277=992f98999ce7b2c13e6adb6eca28ff70-ebd1b06b1c10407b903431c38f993f0f 2023-02-13 21:57:57,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 251bc68b009792a14e42934b238a6375 2023-02-13 21:57:57,564 INFO [StoreOpener-992f98999ce7b2c13e6adb6eca28ff70-1] regionserver.HStore(310): Store=992f98999ce7b2c13e6adb6eca28ff70/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:57,564 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:57,565 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:57,566 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:57:57,567 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:57:57,567 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-13 21:57:57,568 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 4e757d660af6a5aaa6e2c43f8b91311d; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67281478, jitterRate=0.0025721490383148193}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:57,568 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 4e757d660af6a5aaa6e2c43f8b91311d: Running coprocessor pre-open hook at 1676325477540Writing region info on filesystem at 1676325477540Initializing all the Stores at 1676325477541 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325477541Cleaning up temporary data from old regions at 1676325477564 (+23 ms)Cleaning up detritus from prior splits at 1676325477564Running coprocessor post-open hooks at 1676325477568 (+4 ms)Region opened successfully at 1676325477568 2023-02-13 21:57:57,568 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-13 21:57:57,568 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 251bc68b009792a14e42934b238a6375; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69156225, jitterRate=0.03050805628299713}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:57,568 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d., pid=185, masterSystemTime=1676325477536 2023-02-13 21:57:57,569 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 251bc68b009792a14e42934b238a6375: Running coprocessor pre-open hook at 1676325477536Writing region info on filesystem at 1676325477536Initializing all the Stores at 1676325477536Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325477536Cleaning up temporary data from old regions at 1676325477561 (+25 ms)Cleaning up detritus from prior splits at 1676325477562 (+1 ms)Running coprocessor post-open hooks at 1676325477569 (+7 ms)Region opened successfully at 1676325477569 2023-02-13 21:57:57,569 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-13 21:57:57,569 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375., pid=182, masterSystemTime=1676325477532 2023-02-13 21:57:57,569 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 992f98999ce7b2c13e6adb6eca28ff70; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60243443, jitterRate=-0.10230274498462677}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:57,569 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 992f98999ce7b2c13e6adb6eca28ff70: Running coprocessor pre-open hook at 1676325477537Writing region info on filesystem at 1676325477537Initializing all the Stores at 1676325477538 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325477538Cleaning up temporary data from old regions at 1676325477565 (+27 ms)Cleaning up detritus from prior splits at 1676325477565Running coprocessor post-open hooks at 1676325477569 (+4 ms)Region opened successfully at 1676325477569 2023-02-13 21:57:57,570 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:57,570 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70., pid=186, masterSystemTime=1676325477534 2023-02-13 21:57:57,570 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:57:57,570 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:57,570 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=4e757d660af6a5aaa6e2c43f8b91311d, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:57,570 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:57,570 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:57:57,570 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 44e821dce55cb29b05989bca574cff62, NAME => 'testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 21:57:57,570 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:57,571 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1f6dc79bf11da8444a0870d779d2fb4d, NAME => 'testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 21:57:57,571 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:57,571 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:57,571 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:57,571 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:57,571 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:57,570 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325477570"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325477570"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325477570"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325477570"}]},"ts":"1676325477570"} 2023-02-13 21:57:57,571 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:57,571 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:57,571 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:57,572 INFO [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:57,572 INFO [StoreOpener-44e821dce55cb29b05989bca574cff62-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:57,572 DEBUG [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf 2023-02-13 21:57:57,572 DEBUG [StoreOpener-44e821dce55cb29b05989bca574cff62-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf 2023-02-13 21:57:57,573 DEBUG [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf 2023-02-13 21:57:57,573 INFO [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1f6dc79bf11da8444a0870d779d2fb4d columnFamilyName cf 2023-02-13 21:57:57,574 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=251bc68b009792a14e42934b238a6375, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:57,574 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325477573"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325477573"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325477573"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325477573"}]},"ts":"1676325477573"} 2023-02-13 21:57:57,573 DEBUG [StoreOpener-44e821dce55cb29b05989bca574cff62-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf 2023-02-13 21:57:57,574 INFO [StoreOpener-44e821dce55cb29b05989bca574cff62-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 44e821dce55cb29b05989bca574cff62 columnFamilyName cf 2023-02-13 21:57:57,575 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:57,575 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:57:57,575 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:57,575 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 676fb98d6cc62f0fdbda4ce59aad85cb, NAME => 'testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 21:57:57,575 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=992f98999ce7b2c13e6adb6eca28ff70, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:57,575 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325477575"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325477575"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325477575"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325477575"}]},"ts":"1676325477575"} 2023-02-13 21:57:57,575 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676325454277 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:57,575 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=185, resume processing ppid=180 2023-02-13 21:57:57,576 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=185, ppid=180, state=SUCCESS; OpenRegionProcedure 4e757d660af6a5aaa6e2c43f8b91311d, server=jenkins-hbase20.apache.org,36339,1676325451407 in 189 msec 2023-02-13 21:57:57,576 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:57:57,576 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:57,576 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:57,578 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=180, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, ASSIGN in 357 msec 2023-02-13 21:57:57,578 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=182, resume processing ppid=181 2023-02-13 21:57:57,578 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=182, ppid=181, state=SUCCESS; OpenRegionProcedure 251bc68b009792a14e42934b238a6375, server=jenkins-hbase20.apache.org,45015,1676325451448 in 197 msec 2023-02-13 21:57:57,579 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=186, resume processing ppid=177 2023-02-13 21:57:57,579 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=181, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, ASSIGN in 359 msec 2023-02-13 21:57:57,579 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=186, ppid=177, state=SUCCESS; OpenRegionProcedure 992f98999ce7b2c13e6adb6eca28ff70, server=jenkins-hbase20.apache.org,39523,1676325451480 in 192 msec 2023-02-13 21:57:57,580 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=177, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, ASSIGN in 360 msec 2023-02-13 21:57:57,581 INFO [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:57,582 DEBUG [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf 2023-02-13 21:57:57,582 DEBUG [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf 2023-02-13 21:57:57,582 INFO [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 676fb98d6cc62f0fdbda4ce59aad85cb columnFamilyName cf 2023-02-13 21:57:57,589 DEBUG [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/testRestoreSnapshot-1676325454277=1f6dc79bf11da8444a0870d779d2fb4d-3d85d09beeee48a5b67b39f0abb4165e 2023-02-13 21:57:57,590 DEBUG [StoreOpener-44e821dce55cb29b05989bca574cff62-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/testRestoreSnapshot-1676325454277=44e821dce55cb29b05989bca574cff62-8795edd282b341fba51b1c197d77b4f9 2023-02-13 21:57:57,592 DEBUG [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/testRestoreSnapshot-1676325454277=676fb98d6cc62f0fdbda4ce59aad85cb-24e696c302c64bd2bfbbbddca37e4305 2023-02-13 21:57:57,593 DEBUG [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/testRestoreSnapshot-1676325454277=1f6dc79bf11da8444a0870d779d2fb4d-ecd3475741414bf0a2161d2b27bc209a 2023-02-13 21:57:57,593 DEBUG [StoreOpener-44e821dce55cb29b05989bca574cff62-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/testRestoreSnapshot-1676325454277=44e821dce55cb29b05989bca574cff62-f9ada1ebb3d0446a9743d17910965899 2023-02-13 21:57:57,593 INFO [StoreOpener-1f6dc79bf11da8444a0870d779d2fb4d-1] regionserver.HStore(310): Store=1f6dc79bf11da8444a0870d779d2fb4d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:57,593 INFO [StoreOpener-44e821dce55cb29b05989bca574cff62-1] regionserver.HStore(310): Store=44e821dce55cb29b05989bca574cff62/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:57,594 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:57,594 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:57,594 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:57,594 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:57,596 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 44e821dce55cb29b05989bca574cff62 2023-02-13 21:57:57,596 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:57:57,598 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-13 21:57:57,598 DEBUG [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/testRestoreSnapshot-1676325454277=676fb98d6cc62f0fdbda4ce59aad85cb-99375a84534848d588fa45ac3699c454 2023-02-13 21:57:57,598 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 1f6dc79bf11da8444a0870d779d2fb4d; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73695061, jitterRate=0.09814198315143585}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:57,598 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 1f6dc79bf11da8444a0870d779d2fb4d: Running coprocessor pre-open hook at 1676325477571Writing region info on filesystem at 1676325477571Initializing all the Stores at 1676325477571Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325477572 (+1 ms)Cleaning up temporary data from old regions at 1676325477595 (+23 ms)Cleaning up detritus from prior splits at 1676325477595Running coprocessor post-open hooks at 1676325477598 (+3 ms)Region opened successfully at 1676325477598 2023-02-13 21:57:57,598 INFO [StoreOpener-676fb98d6cc62f0fdbda4ce59aad85cb-1] regionserver.HStore(310): Store=676fb98d6cc62f0fdbda4ce59aad85cb/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:57:57,599 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-13 21:57:57,599 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d., pid=183, masterSystemTime=1676325477532 2023-02-13 21:57:57,599 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:57,600 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 44e821dce55cb29b05989bca574cff62; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68590793, jitterRate=0.022082462906837463}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:57,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:57,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 44e821dce55cb29b05989bca574cff62: Running coprocessor pre-open hook at 1676325477571Writing region info on filesystem at 1676325477571Initializing all the Stores at 1676325477572 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325477572Cleaning up temporary data from old regions at 1676325477595 (+23 ms)Cleaning up detritus from prior splits at 1676325477595Running coprocessor post-open hooks at 1676325477600 (+5 ms)Region opened successfully at 1676325477600 2023-02-13 21:57:57,601 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62., pid=187, masterSystemTime=1676325477536 2023-02-13 21:57:57,602 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:57:57,604 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:57,604 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:57:57,604 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-13 21:57:57,604 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=1f6dc79bf11da8444a0870d779d2fb4d, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:57:57,604 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:57,604 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325477604"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325477604"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325477604"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325477604"}]},"ts":"1676325477604"} 2023-02-13 21:57:57,604 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:57:57,605 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=44e821dce55cb29b05989bca574cff62, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:57:57,605 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 676fb98d6cc62f0fdbda4ce59aad85cb; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72038122, jitterRate=0.07345166802406311}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:57:57,605 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325477605"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325477605"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325477605"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325477605"}]},"ts":"1676325477605"} 2023-02-13 21:57:57,605 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 676fb98d6cc62f0fdbda4ce59aad85cb: Running coprocessor pre-open hook at 1676325477576Writing region info on filesystem at 1676325477576Initializing all the Stores at 1676325477577 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325477577Cleaning up temporary data from old regions at 1676325477600 (+23 ms)Cleaning up detritus from prior splits at 1676325477600Running coprocessor post-open hooks at 1676325477605 (+5 ms)Region opened successfully at 1676325477605 2023-02-13 21:57:57,605 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb., pid=184, masterSystemTime=1676325477534 2023-02-13 21:57:57,606 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:57,606 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:57:57,607 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=676fb98d6cc62f0fdbda4ce59aad85cb, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:57:57,607 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325477606"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325477606"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325477606"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325477606"}]},"ts":"1676325477606"} 2023-02-13 21:57:57,607 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=183, resume processing ppid=176 2023-02-13 21:57:57,607 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=183, ppid=176, state=SUCCESS; OpenRegionProcedure 1f6dc79bf11da8444a0870d779d2fb4d, server=jenkins-hbase20.apache.org,45015,1676325451448 in 225 msec 2023-02-13 21:57:57,607 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=187, resume processing ppid=178 2023-02-13 21:57:57,608 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=187, ppid=178, state=SUCCESS; OpenRegionProcedure 44e821dce55cb29b05989bca574cff62, server=jenkins-hbase20.apache.org,36339,1676325451407 in 219 msec 2023-02-13 21:57:57,611 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=176, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, ASSIGN in 388 msec 2023-02-13 21:57:57,611 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=178, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, ASSIGN in 389 msec 2023-02-13 21:57:57,611 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=184, resume processing ppid=179 2023-02-13 21:57:57,611 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=184, ppid=179, state=SUCCESS; OpenRegionProcedure 676fb98d6cc62f0fdbda4ce59aad85cb, server=jenkins-hbase20.apache.org,39523,1676325451480 in 225 msec 2023-02-13 21:57:57,613 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=179, resume processing ppid=175 2023-02-13 21:57:57,613 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=179, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, ASSIGN in 392 msec 2023-02-13 21:57:57,613 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325477613"}]},"ts":"1676325477613"} 2023-02-13 21:57:57,614 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676325454277, state=ENABLED in hbase:meta 2023-02-13 21:57:57,618 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=175, state=SUCCESS; CloneSnapshotProcedure (table=testRestoreSnapshot-1676325454277 snapshot=name: "snaptb1-1676325454277" table: "testRestoreSnapshot-1676325454277" creation_time: 1676325465457 type: DISABLED version: 2 ttl: 0 ) in 521 msec 2023-02-13 21:57:57,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-13 21:57:57,857 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676325454277, procId: 175 completed 2023-02-13 21:57:57,858 DEBUG [Listener at localhost.localdomain/44075] zookeeper.ReadOnlyZKClient(139): Connect 0x79efc34c to 127.0.0.1:58580 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 21:57:57,869 DEBUG [Listener at localhost.localdomain/44075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4dd35f79, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 21:57:57,873 DEBUG [hconnection-0x383008f9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 21:57:57,874 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42490, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 21:57:57,880 DEBUG [hconnection-0x383008f9-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 21:57:57,881 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:51150, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 21:57:57,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:57,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:57,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:57,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:57,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:57,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:57,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:57,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:57,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:57,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:57,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:57,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:57,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:57,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:57,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:57,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:57,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:57,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:57,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:57,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:57,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:57,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:57,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:57,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:57,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:57,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:57,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:57,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:57,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,153 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 21:57:58,157 DEBUG [hconnection-0x383008f9-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 21:57:58,158 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55866, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 21:57:58,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,300 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x27ddad26 to 127.0.0.1:58580 2023-02-13 21:57:58,300 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 21:57:58,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,421 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 21:57:58,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,634 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 21:57:58,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,865 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 21:57:58,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:58,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,107 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 21:57:59,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:57:59,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:00,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:01,554 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676325454277 2023-02-13 21:58:01,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.HMaster$11(2405): Client=jenkins//148.251.75.209 disable testRestoreSnapshot-1676325454277 2023-02-13 21:58:01,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=188, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676325454277 2023-02-13 21:58:01,558 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325481558"}]},"ts":"1676325481558"} 2023-02-13 21:58:01,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-13 21:58:01,559 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676325454277, state=DISABLING in hbase:meta 2023-02-13 21:58:01,560 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676325454277 to state=DISABLING 2023-02-13 21:58:01,562 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, UNASSIGN}, {pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, UNASSIGN}, {pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, UNASSIGN}, {pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, UNASSIGN}, {pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, UNASSIGN}, {pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, UNASSIGN}] 2023-02-13 21:58:01,563 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, UNASSIGN 2023-02-13 21:58:01,564 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, UNASSIGN 2023-02-13 21:58:01,564 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, UNASSIGN 2023-02-13 21:58:01,564 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, UNASSIGN 2023-02-13 21:58:01,564 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, UNASSIGN 2023-02-13 21:58:01,564 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=1f6dc79bf11da8444a0870d779d2fb4d, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:58:01,564 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=251bc68b009792a14e42934b238a6375, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:58:01,564 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=44e821dce55cb29b05989bca574cff62, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:58:01,565 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325481564"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325481564"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325481564"}]},"ts":"1676325481564"} 2023-02-13 21:58:01,564 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=676fb98d6cc62f0fdbda4ce59aad85cb, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:58:01,564 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=4e757d660af6a5aaa6e2c43f8b91311d, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:58:01,565 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325481564"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325481564"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325481564"}]},"ts":"1676325481564"} 2023-02-13 21:58:01,565 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325481564"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325481564"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325481564"}]},"ts":"1676325481564"} 2023-02-13 21:58:01,565 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325481564"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325481564"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325481564"}]},"ts":"1676325481564"} 2023-02-13 21:58:01,565 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325481564"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325481564"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325481564"}]},"ts":"1676325481564"} 2023-02-13 21:58:01,567 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=195, ppid=194, state=RUNNABLE; CloseRegionProcedure 251bc68b009792a14e42934b238a6375, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:58:01,568 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=196, ppid=192, state=RUNNABLE; CloseRegionProcedure 676fb98d6cc62f0fdbda4ce59aad85cb, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:58:01,568 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=197, ppid=191, state=RUNNABLE; CloseRegionProcedure 44e821dce55cb29b05989bca574cff62, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:58:01,572 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=198, ppid=189, state=RUNNABLE; CloseRegionProcedure 1f6dc79bf11da8444a0870d779d2fb4d, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:58:01,573 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=199, ppid=193, state=RUNNABLE; CloseRegionProcedure 4e757d660af6a5aaa6e2c43f8b91311d, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:58:01,573 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, UNASSIGN 2023-02-13 21:58:01,575 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=992f98999ce7b2c13e6adb6eca28ff70, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:58:01,575 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325481575"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325481575"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325481575"}]},"ts":"1676325481575"} 2023-02-13 21:58:01,579 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=200, ppid=190, state=RUNNABLE; CloseRegionProcedure 992f98999ce7b2c13e6adb6eca28ff70, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:58:01,719 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:58:01,720 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 1f6dc79bf11da8444a0870d779d2fb4d, disabling compactions & flushes 2023-02-13 21:58:01,720 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:58:01,720 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:58:01,721 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. after waiting 0 ms 2023-02-13 21:58:01,721 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:58:01,725 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:58:01,725 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:58:01,726 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 676fb98d6cc62f0fdbda4ce59aad85cb, disabling compactions & flushes 2023-02-13 21:58:01,726 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 4e757d660af6a5aaa6e2c43f8b91311d, disabling compactions & flushes 2023-02-13 21:58:01,726 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:58:01,726 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:58:01,726 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:58:01,726 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. after waiting 0 ms 2023-02-13 21:58:01,726 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:58:01,726 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:58:01,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. after waiting 0 ms 2023-02-13 21:58:01,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:58:01,730 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-13 21:58:01,731 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d. 2023-02-13 21:58:01,731 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 1f6dc79bf11da8444a0870d779d2fb4d: Waiting for close lock at 1676325481720Running coprocessor pre-close hooks at 1676325481720Disabling compacts and flushes for region at 1676325481720Disabling writes for close at 1676325481721 (+1 ms)Writing region close event to WAL at 1676325481725 (+4 ms)Running coprocessor post-close hooks at 1676325481730 (+5 ms)Closed at 1676325481731 (+1 ms) 2023-02-13 21:58:01,737 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=1f6dc79bf11da8444a0870d779d2fb4d, regionState=CLOSED 2023-02-13 21:58:01,737 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:58:01,738 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 251bc68b009792a14e42934b238a6375 2023-02-13 21:58:01,738 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325481737"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325481737"}]},"ts":"1676325481737"} 2023-02-13 21:58:01,738 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 251bc68b009792a14e42934b238a6375, disabling compactions & flushes 2023-02-13 21:58:01,739 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:58:01,739 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:58:01,739 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. after waiting 0 ms 2023-02-13 21:58:01,739 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:58:01,742 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=198, resume processing ppid=189 2023-02-13 21:58:01,742 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=198, ppid=189, state=SUCCESS; CloseRegionProcedure 1f6dc79bf11da8444a0870d779d2fb4d, server=jenkins-hbase20.apache.org,45015,1676325451448 in 171 msec 2023-02-13 21:58:01,743 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=189, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=1f6dc79bf11da8444a0870d779d2fb4d, UNASSIGN in 180 msec 2023-02-13 21:58:01,756 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-13 21:58:01,762 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb. 2023-02-13 21:58:01,762 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 676fb98d6cc62f0fdbda4ce59aad85cb: Waiting for close lock at 1676325481725Running coprocessor pre-close hooks at 1676325481725Disabling compacts and flushes for region at 1676325481725Disabling writes for close at 1676325481727 (+2 ms)Writing region close event to WAL at 1676325481735 (+8 ms)Running coprocessor post-close hooks at 1676325481762 (+27 ms)Closed at 1676325481762 2023-02-13 21:58:01,756 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-13 21:58:01,763 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d. 2023-02-13 21:58:01,763 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 4e757d660af6a5aaa6e2c43f8b91311d: Waiting for close lock at 1676325481725Running coprocessor pre-close hooks at 1676325481725Disabling compacts and flushes for region at 1676325481725Disabling writes for close at 1676325481726 (+1 ms)Writing region close event to WAL at 1676325481734 (+8 ms)Running coprocessor post-close hooks at 1676325481763 (+29 ms)Closed at 1676325481763 2023-02-13 21:58:01,764 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:58:01,764 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:58:01,764 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=676fb98d6cc62f0fdbda4ce59aad85cb, regionState=CLOSED 2023-02-13 21:58:01,765 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 992f98999ce7b2c13e6adb6eca28ff70, disabling compactions & flushes 2023-02-13 21:58:01,766 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:58:01,766 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325481764"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325481764"}]},"ts":"1676325481764"} 2023-02-13 21:58:01,766 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:58:01,766 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. after waiting 0 ms 2023-02-13 21:58:01,766 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:58:01,767 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:58:01,767 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 44e821dce55cb29b05989bca574cff62 2023-02-13 21:58:01,768 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 44e821dce55cb29b05989bca574cff62, disabling compactions & flushes 2023-02-13 21:58:01,768 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:58:01,768 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:58:01,768 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. after waiting 0 ms 2023-02-13 21:58:01,768 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:58:01,769 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=4e757d660af6a5aaa6e2c43f8b91311d, regionState=CLOSED 2023-02-13 21:58:01,769 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325481769"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325481769"}]},"ts":"1676325481769"} 2023-02-13 21:58:01,772 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=196, resume processing ppid=192 2023-02-13 21:58:01,773 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=196, ppid=192, state=SUCCESS; CloseRegionProcedure 676fb98d6cc62f0fdbda4ce59aad85cb, server=jenkins-hbase20.apache.org,39523,1676325451480 in 203 msec 2023-02-13 21:58:01,773 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=192, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=676fb98d6cc62f0fdbda4ce59aad85cb, UNASSIGN in 210 msec 2023-02-13 21:58:01,775 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-13 21:58:01,775 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=199, resume processing ppid=193 2023-02-13 21:58:01,776 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=199, ppid=193, state=SUCCESS; CloseRegionProcedure 4e757d660af6a5aaa6e2c43f8b91311d, server=jenkins-hbase20.apache.org,36339,1676325451407 in 201 msec 2023-02-13 21:58:01,776 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=193, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=4e757d660af6a5aaa6e2c43f8b91311d, UNASSIGN in 214 msec 2023-02-13 21:58:01,779 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70. 2023-02-13 21:58:01,779 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 992f98999ce7b2c13e6adb6eca28ff70: Waiting for close lock at 1676325481764Running coprocessor pre-close hooks at 1676325481764Disabling compacts and flushes for region at 1676325481764Disabling writes for close at 1676325481766 (+2 ms)Writing region close event to WAL at 1676325481767 (+1 ms)Running coprocessor post-close hooks at 1676325481779 (+12 ms)Closed at 1676325481779 2023-02-13 21:58:01,781 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:58:01,781 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=992f98999ce7b2c13e6adb6eca28ff70, regionState=CLOSED 2023-02-13 21:58:01,781 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325481781"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325481781"}]},"ts":"1676325481781"} 2023-02-13 21:58:01,783 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-13 21:58:01,783 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-13 21:58:01,783 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62. 2023-02-13 21:58:01,783 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 44e821dce55cb29b05989bca574cff62: Waiting for close lock at 1676325481767Running coprocessor pre-close hooks at 1676325481767Disabling compacts and flushes for region at 1676325481767Disabling writes for close at 1676325481768 (+1 ms)Writing region close event to WAL at 1676325481771 (+3 ms)Running coprocessor post-close hooks at 1676325481783 (+12 ms)Closed at 1676325481783 2023-02-13 21:58:01,784 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=200, resume processing ppid=190 2023-02-13 21:58:01,784 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=200, ppid=190, state=SUCCESS; CloseRegionProcedure 992f98999ce7b2c13e6adb6eca28ff70, server=jenkins-hbase20.apache.org,39523,1676325451480 in 203 msec 2023-02-13 21:58:01,784 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375. 2023-02-13 21:58:01,784 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 251bc68b009792a14e42934b238a6375: Waiting for close lock at 1676325481738Running coprocessor pre-close hooks at 1676325481738Disabling compacts and flushes for region at 1676325481738Disabling writes for close at 1676325481739 (+1 ms)Writing region close event to WAL at 1676325481763 (+24 ms)Running coprocessor post-close hooks at 1676325481784 (+21 ms)Closed at 1676325481784 2023-02-13 21:58:01,784 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=190, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=992f98999ce7b2c13e6adb6eca28ff70, UNASSIGN in 222 msec 2023-02-13 21:58:01,784 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 44e821dce55cb29b05989bca574cff62 2023-02-13 21:58:01,785 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=44e821dce55cb29b05989bca574cff62, regionState=CLOSED 2023-02-13 21:58:01,785 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676325481785"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325481785"}]},"ts":"1676325481785"} 2023-02-13 21:58:01,785 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 251bc68b009792a14e42934b238a6375 2023-02-13 21:58:01,785 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=251bc68b009792a14e42934b238a6375, regionState=CLOSED 2023-02-13 21:58:01,786 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676325481785"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325481785"}]},"ts":"1676325481785"} 2023-02-13 21:58:01,788 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=197, resume processing ppid=191 2023-02-13 21:58:01,788 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=197, ppid=191, state=SUCCESS; CloseRegionProcedure 44e821dce55cb29b05989bca574cff62, server=jenkins-hbase20.apache.org,36339,1676325451407 in 218 msec 2023-02-13 21:58:01,788 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=195, resume processing ppid=194 2023-02-13 21:58:01,788 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=195, ppid=194, state=SUCCESS; CloseRegionProcedure 251bc68b009792a14e42934b238a6375, server=jenkins-hbase20.apache.org,45015,1676325451448 in 221 msec 2023-02-13 21:58:01,789 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=191, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=44e821dce55cb29b05989bca574cff62, UNASSIGN in 226 msec 2023-02-13 21:58:01,789 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=194, resume processing ppid=188 2023-02-13 21:58:01,789 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=194, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676325454277, region=251bc68b009792a14e42934b238a6375, UNASSIGN in 226 msec 2023-02-13 21:58:01,790 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325481790"}]},"ts":"1676325481790"} 2023-02-13 21:58:01,791 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676325454277, state=DISABLED in hbase:meta 2023-02-13 21:58:01,795 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676325454277 to state=DISABLED 2023-02-13 21:58:01,797 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=188, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676325454277 in 242 msec 2023-02-13 21:58:01,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-13 21:58:01,809 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676325454277, procId: 188 completed 2023-02-13 21:58:01,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.HMaster$5(2215): Client=jenkins//148.251.75.209 delete testRestoreSnapshot-1676325454277 2023-02-13 21:58:01,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1676325454277 2023-02-13 21:58:01,812 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676325454277 2023-02-13 21:58:01,813 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=201, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676325454277 2023-02-13 21:58:01,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-02-13 21:58:01,820 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:58:01,820 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:58:01,846 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62 2023-02-13 21:58:01,846 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:58:01,846 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:58:01,846 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375 2023-02-13 21:58:01,848 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x79efc34c to 127.0.0.1:58580 2023-02-13 21:58:01,848 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 21:58:01,853 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf, FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/recovered.edits] 2023-02-13 21:58:01,854 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf, FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/recovered.edits] 2023-02-13 21:58:01,854 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf, FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/recovered.edits] 2023-02-13 21:58:01,854 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf, FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/recovered.edits] 2023-02-13 21:58:01,854 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf, FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/recovered.edits] 2023-02-13 21:58:01,856 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf, FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/recovered.edits] 2023-02-13 21:58:01,857 ERROR [HFileArchiver-5] backup.HFileArchiver(523): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/testRestoreSnapshot-1676325454277=4e757d660af6a5aaa6e2c43f8b91311d-08e8b394e04742929b62615c32c8ed4c already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/testRestoreSnapshot-1676325454277=4e757d660af6a5aaa6e2c43f8b91311d-08e8b394e04742929b62615c32c8ed4c. archiveLen: 0 currentLen: 0 archiveMtime: 1676325476832 currentMtime: 1676325477204 2023-02-13 21:58:01,858 INFO [HFileArchiver-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/testRestoreSnapshot-1676325454277=4e757d660af6a5aaa6e2c43f8b91311d-08e8b394e04742929b62615c32c8ed4c to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/testRestoreSnapshot-1676325454277=4e757d660af6a5aaa6e2c43f8b91311d-08e8b394e04742929b62615c32c8ed4c.1676325481854. 2023-02-13 21:58:01,858 ERROR [HFileArchiver-1] backup.HFileArchiver(523): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/testRestoreSnapshot-1676325454277=44e821dce55cb29b05989bca574cff62-8795edd282b341fba51b1c197d77b4f9 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/testRestoreSnapshot-1676325454277=44e821dce55cb29b05989bca574cff62-8795edd282b341fba51b1c197d77b4f9. archiveLen: 0 currentLen: 0 archiveMtime: 1676325476832 currentMtime: 1676325477190 2023-02-13 21:58:01,858 ERROR [HFileArchiver-3] backup.HFileArchiver(523): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/testRestoreSnapshot-1676325454277=251bc68b009792a14e42934b238a6375-136600ed36234d64b213b50e0d6f02a9 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/testRestoreSnapshot-1676325454277=251bc68b009792a14e42934b238a6375-136600ed36234d64b213b50e0d6f02a9. archiveLen: 0 currentLen: 0 archiveMtime: 1676325476832 currentMtime: 1676325477202 2023-02-13 21:58:01,858 INFO [HFileArchiver-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/testRestoreSnapshot-1676325454277=44e821dce55cb29b05989bca574cff62-8795edd282b341fba51b1c197d77b4f9 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/testRestoreSnapshot-1676325454277=44e821dce55cb29b05989bca574cff62-8795edd282b341fba51b1c197d77b4f9.1676325481854. 2023-02-13 21:58:01,858 ERROR [HFileArchiver-2] backup.HFileArchiver(523): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/testRestoreSnapshot-1676325454277=1f6dc79bf11da8444a0870d779d2fb4d-3d85d09beeee48a5b67b39f0abb4165e already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/testRestoreSnapshot-1676325454277=1f6dc79bf11da8444a0870d779d2fb4d-3d85d09beeee48a5b67b39f0abb4165e. archiveLen: 0 currentLen: 0 archiveMtime: 1676325476832 currentMtime: 1676325477189 2023-02-13 21:58:01,859 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/testRestoreSnapshot-1676325454277=4e757d660af6a5aaa6e2c43f8b91311d-08e8b394e04742929b62615c32c8ed4c to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/testRestoreSnapshot-1676325454277=4e757d660af6a5aaa6e2c43f8b91311d-08e8b394e04742929b62615c32c8ed4c 2023-02-13 21:58:01,859 INFO [HFileArchiver-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/testRestoreSnapshot-1676325454277=251bc68b009792a14e42934b238a6375-136600ed36234d64b213b50e0d6f02a9 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/testRestoreSnapshot-1676325454277=251bc68b009792a14e42934b238a6375-136600ed36234d64b213b50e0d6f02a9.1676325481853. 2023-02-13 21:58:01,859 ERROR [HFileArchiver-6] backup.HFileArchiver(523): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/testRestoreSnapshot-1676325454277=992f98999ce7b2c13e6adb6eca28ff70-46bad1cc5702482388220a80e425cbe0 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/testRestoreSnapshot-1676325454277=992f98999ce7b2c13e6adb6eca28ff70-46bad1cc5702482388220a80e425cbe0. archiveLen: 0 currentLen: 0 archiveMtime: 1676325476833 currentMtime: 1676325477199 2023-02-13 21:58:01,859 INFO [HFileArchiver-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/testRestoreSnapshot-1676325454277=1f6dc79bf11da8444a0870d779d2fb4d-3d85d09beeee48a5b67b39f0abb4165e to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/testRestoreSnapshot-1676325454277=1f6dc79bf11da8444a0870d779d2fb4d-3d85d09beeee48a5b67b39f0abb4165e.1676325481854. 2023-02-13 21:58:01,859 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/testRestoreSnapshot-1676325454277=44e821dce55cb29b05989bca574cff62-8795edd282b341fba51b1c197d77b4f9 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/testRestoreSnapshot-1676325454277=44e821dce55cb29b05989bca574cff62-8795edd282b341fba51b1c197d77b4f9 2023-02-13 21:58:01,859 INFO [HFileArchiver-6] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/testRestoreSnapshot-1676325454277=992f98999ce7b2c13e6adb6eca28ff70-46bad1cc5702482388220a80e425cbe0 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/testRestoreSnapshot-1676325454277=992f98999ce7b2c13e6adb6eca28ff70-46bad1cc5702482388220a80e425cbe0.1676325481854. 2023-02-13 21:58:01,860 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/testRestoreSnapshot-1676325454277=251bc68b009792a14e42934b238a6375-136600ed36234d64b213b50e0d6f02a9 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/testRestoreSnapshot-1676325454277=251bc68b009792a14e42934b238a6375-136600ed36234d64b213b50e0d6f02a9 2023-02-13 21:58:01,860 ERROR [HFileArchiver-7] backup.HFileArchiver(523): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/testRestoreSnapshot-1676325454277=676fb98d6cc62f0fdbda4ce59aad85cb-24e696c302c64bd2bfbbbddca37e4305 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/testRestoreSnapshot-1676325454277=676fb98d6cc62f0fdbda4ce59aad85cb-24e696c302c64bd2bfbbbddca37e4305. archiveLen: 0 currentLen: 0 archiveMtime: 1676325476831 currentMtime: 1676325477202 2023-02-13 21:58:01,860 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/testRestoreSnapshot-1676325454277=1f6dc79bf11da8444a0870d779d2fb4d-3d85d09beeee48a5b67b39f0abb4165e to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/testRestoreSnapshot-1676325454277=1f6dc79bf11da8444a0870d779d2fb4d-3d85d09beeee48a5b67b39f0abb4165e 2023-02-13 21:58:01,861 INFO [HFileArchiver-7] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/testRestoreSnapshot-1676325454277=676fb98d6cc62f0fdbda4ce59aad85cb-24e696c302c64bd2bfbbbddca37e4305 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/testRestoreSnapshot-1676325454277=676fb98d6cc62f0fdbda4ce59aad85cb-24e696c302c64bd2bfbbbddca37e4305.1676325481856. 2023-02-13 21:58:01,861 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/testRestoreSnapshot-1676325454277=992f98999ce7b2c13e6adb6eca28ff70-46bad1cc5702482388220a80e425cbe0 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/testRestoreSnapshot-1676325454277=992f98999ce7b2c13e6adb6eca28ff70-46bad1cc5702482388220a80e425cbe0 2023-02-13 21:58:01,861 ERROR [HFileArchiver-5] backup.HFileArchiver(523): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/testRestoreSnapshot-1676325454277=4e757d660af6a5aaa6e2c43f8b91311d-81bb121c83f84b0997b6675992ad85a1 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/testRestoreSnapshot-1676325454277=4e757d660af6a5aaa6e2c43f8b91311d-81bb121c83f84b0997b6675992ad85a1. archiveLen: 0 currentLen: 0 archiveMtime: 1676325476833 currentMtime: 1676325477206 2023-02-13 21:58:01,861 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/testRestoreSnapshot-1676325454277=676fb98d6cc62f0fdbda4ce59aad85cb-24e696c302c64bd2bfbbbddca37e4305 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/testRestoreSnapshot-1676325454277=676fb98d6cc62f0fdbda4ce59aad85cb-24e696c302c64bd2bfbbbddca37e4305 2023-02-13 21:58:01,862 ERROR [HFileArchiver-1] backup.HFileArchiver(523): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/testRestoreSnapshot-1676325454277=44e821dce55cb29b05989bca574cff62-f9ada1ebb3d0446a9743d17910965899 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/testRestoreSnapshot-1676325454277=44e821dce55cb29b05989bca574cff62-f9ada1ebb3d0446a9743d17910965899. archiveLen: 0 currentLen: 0 archiveMtime: 1676325476835 currentMtime: 1676325477202 2023-02-13 21:58:01,862 ERROR [HFileArchiver-3] backup.HFileArchiver(523): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/testRestoreSnapshot-1676325454277=251bc68b009792a14e42934b238a6375-29f391c114814c4b96610425ae0e5de0 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/testRestoreSnapshot-1676325454277=251bc68b009792a14e42934b238a6375-29f391c114814c4b96610425ae0e5de0. archiveLen: 0 currentLen: 0 archiveMtime: 1676325476834 currentMtime: 1676325477205 2023-02-13 21:58:01,862 INFO [HFileArchiver-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/testRestoreSnapshot-1676325454277=4e757d660af6a5aaa6e2c43f8b91311d-81bb121c83f84b0997b6675992ad85a1 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/testRestoreSnapshot-1676325454277=4e757d660af6a5aaa6e2c43f8b91311d-81bb121c83f84b0997b6675992ad85a1.1676325481854. 2023-02-13 21:58:01,863 ERROR [HFileArchiver-2] backup.HFileArchiver(523): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/testRestoreSnapshot-1676325454277=1f6dc79bf11da8444a0870d779d2fb4d-ecd3475741414bf0a2161d2b27bc209a already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/testRestoreSnapshot-1676325454277=1f6dc79bf11da8444a0870d779d2fb4d-ecd3475741414bf0a2161d2b27bc209a. archiveLen: 0 currentLen: 0 archiveMtime: 1676325476835 currentMtime: 1676325477199 2023-02-13 21:58:01,863 INFO [HFileArchiver-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/testRestoreSnapshot-1676325454277=251bc68b009792a14e42934b238a6375-29f391c114814c4b96610425ae0e5de0 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/testRestoreSnapshot-1676325454277=251bc68b009792a14e42934b238a6375-29f391c114814c4b96610425ae0e5de0.1676325481853. 2023-02-13 21:58:01,863 INFO [HFileArchiver-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/testRestoreSnapshot-1676325454277=44e821dce55cb29b05989bca574cff62-f9ada1ebb3d0446a9743d17910965899 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/testRestoreSnapshot-1676325454277=44e821dce55cb29b05989bca574cff62-f9ada1ebb3d0446a9743d17910965899.1676325481854. 2023-02-13 21:58:01,863 INFO [HFileArchiver-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/testRestoreSnapshot-1676325454277=1f6dc79bf11da8444a0870d779d2fb4d-ecd3475741414bf0a2161d2b27bc209a to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/testRestoreSnapshot-1676325454277=1f6dc79bf11da8444a0870d779d2fb4d-ecd3475741414bf0a2161d2b27bc209a.1676325481854. 2023-02-13 21:58:01,864 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/testRestoreSnapshot-1676325454277=4e757d660af6a5aaa6e2c43f8b91311d-81bb121c83f84b0997b6675992ad85a1 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/cf/testRestoreSnapshot-1676325454277=4e757d660af6a5aaa6e2c43f8b91311d-81bb121c83f84b0997b6675992ad85a1 2023-02-13 21:58:01,864 ERROR [HFileArchiver-6] backup.HFileArchiver(523): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/testRestoreSnapshot-1676325454277=992f98999ce7b2c13e6adb6eca28ff70-ebd1b06b1c10407b903431c38f993f0f already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/testRestoreSnapshot-1676325454277=992f98999ce7b2c13e6adb6eca28ff70-ebd1b06b1c10407b903431c38f993f0f. archiveLen: 0 currentLen: 0 archiveMtime: 1676325476835 currentMtime: 1676325477204 2023-02-13 21:58:01,864 ERROR [HFileArchiver-7] backup.HFileArchiver(523): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/testRestoreSnapshot-1676325454277=676fb98d6cc62f0fdbda4ce59aad85cb-99375a84534848d588fa45ac3699c454 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/testRestoreSnapshot-1676325454277=676fb98d6cc62f0fdbda4ce59aad85cb-99375a84534848d588fa45ac3699c454. archiveLen: 0 currentLen: 0 archiveMtime: 1676325476833 currentMtime: 1676325477205 2023-02-13 21:58:01,864 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/testRestoreSnapshot-1676325454277=1f6dc79bf11da8444a0870d779d2fb4d-ecd3475741414bf0a2161d2b27bc209a to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/cf/testRestoreSnapshot-1676325454277=1f6dc79bf11da8444a0870d779d2fb4d-ecd3475741414bf0a2161d2b27bc209a 2023-02-13 21:58:01,865 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/testRestoreSnapshot-1676325454277=251bc68b009792a14e42934b238a6375-29f391c114814c4b96610425ae0e5de0 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/cf/testRestoreSnapshot-1676325454277=251bc68b009792a14e42934b238a6375-29f391c114814c4b96610425ae0e5de0 2023-02-13 21:58:01,865 INFO [HFileArchiver-6] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/testRestoreSnapshot-1676325454277=992f98999ce7b2c13e6adb6eca28ff70-ebd1b06b1c10407b903431c38f993f0f to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/testRestoreSnapshot-1676325454277=992f98999ce7b2c13e6adb6eca28ff70-ebd1b06b1c10407b903431c38f993f0f.1676325481854. 2023-02-13 21:58:01,865 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/testRestoreSnapshot-1676325454277=44e821dce55cb29b05989bca574cff62-f9ada1ebb3d0446a9743d17910965899 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/cf/testRestoreSnapshot-1676325454277=44e821dce55cb29b05989bca574cff62-f9ada1ebb3d0446a9743d17910965899 2023-02-13 21:58:01,865 INFO [HFileArchiver-7] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/testRestoreSnapshot-1676325454277=676fb98d6cc62f0fdbda4ce59aad85cb-99375a84534848d588fa45ac3699c454 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/testRestoreSnapshot-1676325454277=676fb98d6cc62f0fdbda4ce59aad85cb-99375a84534848d588fa45ac3699c454.1676325481856. 2023-02-13 21:58:01,866 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/testRestoreSnapshot-1676325454277=992f98999ce7b2c13e6adb6eca28ff70-ebd1b06b1c10407b903431c38f993f0f to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/cf/testRestoreSnapshot-1676325454277=992f98999ce7b2c13e6adb6eca28ff70-ebd1b06b1c10407b903431c38f993f0f 2023-02-13 21:58:01,867 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/testRestoreSnapshot-1676325454277=676fb98d6cc62f0fdbda4ce59aad85cb-99375a84534848d588fa45ac3699c454 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/cf/testRestoreSnapshot-1676325454277=676fb98d6cc62f0fdbda4ce59aad85cb-99375a84534848d588fa45ac3699c454 2023-02-13 21:58:01,867 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/recovered.edits/18.seqid to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d/recovered.edits/18.seqid 2023-02-13 21:58:01,869 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/4e757d660af6a5aaa6e2c43f8b91311d 2023-02-13 21:58:01,870 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/recovered.edits/18.seqid to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375/recovered.edits/18.seqid 2023-02-13 21:58:01,870 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/recovered.edits/18.seqid to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62/recovered.edits/18.seqid 2023-02-13 21:58:01,870 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/recovered.edits/18.seqid to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70/recovered.edits/18.seqid 2023-02-13 21:58:01,870 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/recovered.edits/18.seqid to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d/recovered.edits/18.seqid 2023-02-13 21:58:01,870 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/251bc68b009792a14e42934b238a6375 2023-02-13 21:58:01,871 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/44e821dce55cb29b05989bca574cff62 2023-02-13 21:58:01,871 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/992f98999ce7b2c13e6adb6eca28ff70 2023-02-13 21:58:01,871 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/recovered.edits/18.seqid to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb/recovered.edits/18.seqid 2023-02-13 21:58:01,871 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/1f6dc79bf11da8444a0870d779d2fb4d 2023-02-13 21:58:01,871 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testRestoreSnapshot-1676325454277/676fb98d6cc62f0fdbda4ce59aad85cb 2023-02-13 21:58:01,871 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676325454277 regions 2023-02-13 21:58:01,872 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881 2023-02-13 21:58:01,872 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf] 2023-02-13 21:58:01,874 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e. archiveLen: 0 currentLen: 0 archiveMtime: 1676325476854 currentMtime: 1676325477122 2023-02-13 21:58:01,875 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e.1676325481872. 2023-02-13 21:58:01,876 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-a87ff679a2f3e71d9181a67b7542122c20230213c23024dd137c4d878dd3d9a4d7d25e9e 2023-02-13 21:58:01,876 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2. archiveLen: 0 currentLen: 0 archiveMtime: 1676325476855 currentMtime: 1676325477124 2023-02-13 21:58:01,877 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2.1676325481872. 2023-02-13 21:58:01,877 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-a87ff679a2f3e71d9181a67b7542122c20230213ddbafa71b7714482a378b4c797e1dbc2 2023-02-13 21:58:01,878 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1. archiveLen: 0 currentLen: 0 archiveMtime: 1676325476856 currentMtime: 1676325477126 2023-02-13 21:58:01,879 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1.1676325481872. 2023-02-13 21:58:01,879 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c4ca4238a0b923820dcc509a6f75849b20230213355efcf4149a4839ac8fa51e1a749ec1 2023-02-13 21:58:01,880 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8. archiveLen: 0 currentLen: 0 archiveMtime: 1676325476858 currentMtime: 1676325477129 2023-02-13 21:58:01,881 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8.1676325481872. 2023-02-13 21:58:01,881 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c4ca4238a0b923820dcc509a6f75849b20230213ec577643748d41c8bfd77d0e356314d8 2023-02-13 21:58:01,882 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162. archiveLen: 0 currentLen: 0 archiveMtime: 1676325476859 currentMtime: 1676325477131 2023-02-13 21:58:01,883 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162.1676325481872. 2023-02-13 21:58:01,887 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c81e728d9d4c2f636f067f89cc14862c202302134f4db69735504a42bbdbbc1ddc8c6162 2023-02-13 21:58:01,888 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3. archiveLen: 0 currentLen: 0 archiveMtime: 1676325476860 currentMtime: 1676325477133 2023-02-13 21:58:01,888 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3.1676325481872. 2023-02-13 21:58:01,889 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-c81e728d9d4c2f636f067f89cc14862c202302136a246b33d4614c3e9f5a7aaba84dd9f3 2023-02-13 21:58:01,890 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e. archiveLen: 0 currentLen: 0 archiveMtime: 1676325476860 currentMtime: 1676325477135 2023-02-13 21:58:01,891 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e.1676325481872. 2023-02-13 21:58:01,892 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-d41d8cd98f00b204e9800998ecf8427e2023021376b05e26b4cd417ba60e4ec1628c070e 2023-02-13 21:58:01,893 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010. archiveLen: 0 currentLen: 0 archiveMtime: 1676325476861 currentMtime: 1676325477137 2023-02-13 21:58:01,893 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010.1676325481872. 2023-02-13 21:58:01,895 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-d41d8cd98f00b204e9800998ecf8427e20230213e44067e181ce48aa9c03dc605ee4f010 2023-02-13 21:58:01,896 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a. archiveLen: 0 currentLen: 0 archiveMtime: 1676325476862 currentMtime: 1676325477138 2023-02-13 21:58:01,896 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a.1676325481872. 2023-02-13 21:58:01,897 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-e4da3b7fbbce2345d7772b0674a318d52023021394da206f854f46ffbaefc422821cc16a 2023-02-13 21:58:01,898 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683. archiveLen: 0 currentLen: 0 archiveMtime: 1676325476863 currentMtime: 1676325477140 2023-02-13 21:58:01,898 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683.1676325481872. 2023-02-13 21:58:01,899 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-e4da3b7fbbce2345d7772b0674a318d520230213cadd6a8ed3404d0683b766fa95c13683 2023-02-13 21:58:01,900 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3. archiveLen: 0 currentLen: 0 archiveMtime: 1676325476864 currentMtime: 1676325477142 2023-02-13 21:58:01,900 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3.1676325481872. 2023-02-13 21:58:01,901 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-eccbc87e4b5ce2fe28308fd9f2a7baf3202302132913d74358c840ee8d788cacae2d27a3 2023-02-13 21:58:01,903 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b. archiveLen: 0 currentLen: 0 archiveMtime: 1676325476865 currentMtime: 1676325477144 2023-02-13 21:58:01,904 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b.1676325481872. 2023-02-13 21:58:01,905 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881/cf/testRestoreSnapshot-1676325454277=6a0489fa0c27bf9bed60f657753c7881-eccbc87e4b5ce2fe28308fd9f2a7baf3202302135571a73012ff4c65a2953675d3f47b4b 2023-02-13 21:58:01,905 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testRestoreSnapshot-1676325454277/6a0489fa0c27bf9bed60f657753c7881 2023-02-13 21:58:01,908 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=201, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676325454277 2023-02-13 21:58:01,911 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1676325454277 from hbase:meta 2023-02-13 21:58:01,913 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1676325454277' descriptor. 2023-02-13 21:58:01,914 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=201, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676325454277 2023-02-13 21:58:01,914 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1676325454277' from region states. 2023-02-13 21:58:01,914 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325481914"}]},"ts":"9223372036854775807"} 2023-02-13 21:58:01,914 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325481914"}]},"ts":"9223372036854775807"} 2023-02-13 21:58:01,915 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325481914"}]},"ts":"9223372036854775807"} 2023-02-13 21:58:01,915 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325481914"}]},"ts":"9223372036854775807"} 2023-02-13 21:58:01,915 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325481914"}]},"ts":"9223372036854775807"} 2023-02-13 21:58:01,915 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325481914"}]},"ts":"9223372036854775807"} 2023-02-13 21:58:01,918 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-13 21:58:01,918 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 1f6dc79bf11da8444a0870d779d2fb4d, NAME => 'testRestoreSnapshot-1676325454277,,1676325454298.1f6dc79bf11da8444a0870d779d2fb4d.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 992f98999ce7b2c13e6adb6eca28ff70, NAME => 'testRestoreSnapshot-1676325454277,1,1676325454298.992f98999ce7b2c13e6adb6eca28ff70.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 44e821dce55cb29b05989bca574cff62, NAME => 'testRestoreSnapshot-1676325454277,2,1676325454298.44e821dce55cb29b05989bca574cff62.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 676fb98d6cc62f0fdbda4ce59aad85cb, NAME => 'testRestoreSnapshot-1676325454277,3,1676325454298.676fb98d6cc62f0fdbda4ce59aad85cb.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 4e757d660af6a5aaa6e2c43f8b91311d, NAME => 'testRestoreSnapshot-1676325454277,4,1676325454298.4e757d660af6a5aaa6e2c43f8b91311d.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 251bc68b009792a14e42934b238a6375, NAME => 'testRestoreSnapshot-1676325454277,5,1676325454298.251bc68b009792a14e42934b238a6375.', STARTKEY => '5', ENDKEY => ''}] 2023-02-13 21:58:01,918 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1676325454277' as deleted. 2023-02-13 21:58:01,918 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676325454277","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676325481918"}]},"ts":"9223372036854775807"} 2023-02-13 21:58:01,923 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1676325454277 state from META 2023-02-13 21:58:01,925 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=201, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676325454277 2023-02-13 21:58:01,926 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=201, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1676325454277 in 115 msec 2023-02-13 21:58:02,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-02-13 21:58:02,065 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1676325454277, procId: 201 completed 2023-02-13 21:58:02,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(748): Client=jenkins//148.251.75.209 delete name: "emptySnaptb-1676325454277" 2023-02-13 21:58:02,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1676325454277 2023-02-13 21:58:02,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(748): Client=jenkins//148.251.75.209 delete name: "snaptb0-1676325454277" 2023-02-13 21:58:02,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1676325454277 2023-02-13 21:58:02,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(748): Client=jenkins//148.251.75.209 delete name: "snaptb1-1676325454277" 2023-02-13 21:58:02,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(333): Deleting snapshot: snaptb1-1676325454277 2023-02-13 21:58:02,108 INFO [Listener at localhost.localdomain/44075] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=405 (was 363) Potentially hanging thread: hconnection-0x1d9ca414-shared-pool-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1023233772_15 at /127.0.0.1:34140 [Waiting for operation #504] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x43e34ef3-shared-pool-26 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-16 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/cluster_c8eab63c-2879-73ac-9f0f-8254d034d5a9/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-14 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/cluster_c8eab63c-2879-73ac-9f0f-8254d034d5a9/dfs/data/data5/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-15 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x43e34ef3-shared-pool-27 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1023233772_15 at /127.0.0.1:34710 [Waiting for operation #539] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/cluster_c8eab63c-2879-73ac-9f0f-8254d034d5a9/dfs/data/data6/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1023233772_15 at /127.0.0.1:46652 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1023233772_15 at /127.0.0.1:34716 [Waiting for operation #490] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1023233772_15 at /127.0.0.1:34718 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/cluster_c8eab63c-2879-73ac-9f0f-8254d034d5a9/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x43e34ef3-shared-pool-25 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x43e34ef3-shared-pool-28 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/cluster_c8eab63c-2879-73ac-9f0f-8254d034d5a9/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1023233772_15 at /127.0.0.1:34128 [Waiting for operation #483] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/cluster_c8eab63c-2879-73ac-9f0f-8254d034d5a9/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=766 (was 695) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=292 (was 142) - SystemLoadAverage LEAK? -, ProcessCount=164 (was 165), AvailableMemoryMB=6743 (was 7283) 2023-02-13 21:58:02,125 INFO [Listener at localhost.localdomain/44075] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=405, OpenFileDescriptor=766, MaxFileDescriptor=60000, SystemLoadAverage=292, ProcessCount=164, AvailableMemoryMB=6741 2023-02-13 21:58:02,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.HMaster$4(2102): Client=jenkins//148.251.75.209 create 'testCorruptedSnapshot-1676325482126', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-02-13 21:58:02,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testCorruptedSnapshot-1676325482126 2023-02-13 21:58:02,132 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676325482126 execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 21:58:02,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "testCorruptedSnapshot-1676325482126" procId is: 202 2023-02-13 21:58:02,133 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676325482126 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 21:58:02,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-13 21:58:02,138 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78 2023-02-13 21:58:02,138 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292 2023-02-13 21:58:02,138 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f 2023-02-13 21:58:02,138 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40 2023-02-13 21:58:02,138 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487 2023-02-13 21:58:02,138 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9 2023-02-13 21:58:02,139 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78 empty. 2023-02-13 21:58:02,139 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9 empty. 2023-02-13 21:58:02,139 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487 empty. 2023-02-13 21:58:02,139 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f empty. 2023-02-13 21:58:02,139 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40 empty. 2023-02-13 21:58:02,140 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78 2023-02-13 21:58:02,140 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292 empty. 2023-02-13 21:58:02,140 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487 2023-02-13 21:58:02,140 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f 2023-02-13 21:58:02,140 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9 2023-02-13 21:58:02,140 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292 2023-02-13 21:58:02,140 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40 2023-02-13 21:58:02,140 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1676325482126 regions 2023-02-13 21:58:02,157 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/.tabledesc/.tableinfo.0000000001 2023-02-13 21:58:02,158 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-0] regionserver.HRegion(7671): creating {ENCODED => 1c996d2a582a7ec43382d5a39bb54c78, NAME => 'testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testCorruptedSnapshot-1676325482126', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp 2023-02-13 21:58:02,163 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-1] regionserver.HRegion(7671): creating {ENCODED => f244b6e82e7a1752e9a3cb5f28dea2e9, NAME => 'testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testCorruptedSnapshot-1676325482126', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp 2023-02-13 21:58:02,165 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-2] regionserver.HRegion(7671): creating {ENCODED => ac1d4ae96948f4de74aa3114b7308487, NAME => 'testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testCorruptedSnapshot-1676325482126', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp 2023-02-13 21:58:02,183 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:58:02,183 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-0] regionserver.HRegion(1603): Closing 1c996d2a582a7ec43382d5a39bb54c78, disabling compactions & flushes 2023-02-13 21:58:02,183 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. 2023-02-13 21:58:02,183 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. 2023-02-13 21:58:02,183 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. after waiting 0 ms 2023-02-13 21:58:02,183 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. 2023-02-13 21:58:02,183 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. 2023-02-13 21:58:02,183 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-0] regionserver.HRegion(1557): Region close journal for 1c996d2a582a7ec43382d5a39bb54c78: Waiting for close lock at 1676325482183Disabling compacts and flushes for region at 1676325482183Disabling writes for close at 1676325482183Writing region close event to WAL at 1676325482183Closed at 1676325482183 2023-02-13 21:58:02,184 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-0] regionserver.HRegion(7671): creating {ENCODED => d1acdd49740dd716d5bec22786474a40, NAME => 'testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testCorruptedSnapshot-1676325482126', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp 2023-02-13 21:58:02,184 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-1] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:58:02,184 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-2] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:58:02,184 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-1] regionserver.HRegion(1603): Closing f244b6e82e7a1752e9a3cb5f28dea2e9, disabling compactions & flushes 2023-02-13 21:58:02,184 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-2] regionserver.HRegion(1603): Closing ac1d4ae96948f4de74aa3114b7308487, disabling compactions & flushes 2023-02-13 21:58:02,184 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-1] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. 2023-02-13 21:58:02,184 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-2] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. 2023-02-13 21:58:02,184 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. 2023-02-13 21:58:02,184 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. 2023-02-13 21:58:02,184 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-1] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. after waiting 0 ms 2023-02-13 21:58:02,184 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-2] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. after waiting 0 ms 2023-02-13 21:58:02,184 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-1] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. 2023-02-13 21:58:02,184 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-2] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. 2023-02-13 21:58:02,184 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-1] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. 2023-02-13 21:58:02,184 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-2] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. 2023-02-13 21:58:02,185 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-1] regionserver.HRegion(1557): Region close journal for f244b6e82e7a1752e9a3cb5f28dea2e9: Waiting for close lock at 1676325482184Disabling compacts and flushes for region at 1676325482184Disabling writes for close at 1676325482184Writing region close event to WAL at 1676325482184Closed at 1676325482184 2023-02-13 21:58:02,185 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-2] regionserver.HRegion(1557): Region close journal for ac1d4ae96948f4de74aa3114b7308487: Waiting for close lock at 1676325482184Disabling compacts and flushes for region at 1676325482184Disabling writes for close at 1676325482184Writing region close event to WAL at 1676325482184Closed at 1676325482184 2023-02-13 21:58:02,185 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-1] regionserver.HRegion(7671): creating {ENCODED => ef75de132134b515dbe11abe569ac292, NAME => 'testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testCorruptedSnapshot-1676325482126', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp 2023-02-13 21:58:02,185 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-2] regionserver.HRegion(7671): creating {ENCODED => 00d462c2fbf63cefa67cf8e8f681246f, NAME => 'testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testCorruptedSnapshot-1676325482126', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp 2023-02-13 21:58:02,196 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-2] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:58:02,196 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:58:02,196 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-2] regionserver.HRegion(1603): Closing 00d462c2fbf63cefa67cf8e8f681246f, disabling compactions & flushes 2023-02-13 21:58:02,196 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-0] regionserver.HRegion(1603): Closing d1acdd49740dd716d5bec22786474a40, disabling compactions & flushes 2023-02-13 21:58:02,197 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-2] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. 2023-02-13 21:58:02,197 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. 2023-02-13 21:58:02,197 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. 2023-02-13 21:58:02,197 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. 2023-02-13 21:58:02,197 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-2] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. after waiting 0 ms 2023-02-13 21:58:02,197 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-2] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. 2023-02-13 21:58:02,197 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. after waiting 0 ms 2023-02-13 21:58:02,197 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-2] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. 2023-02-13 21:58:02,197 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. 2023-02-13 21:58:02,197 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-2] regionserver.HRegion(1557): Region close journal for 00d462c2fbf63cefa67cf8e8f681246f: Waiting for close lock at 1676325482196Disabling compacts and flushes for region at 1676325482196Disabling writes for close at 1676325482197 (+1 ms)Writing region close event to WAL at 1676325482197Closed at 1676325482197 2023-02-13 21:58:02,197 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. 2023-02-13 21:58:02,197 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-0] regionserver.HRegion(1557): Region close journal for d1acdd49740dd716d5bec22786474a40: Waiting for close lock at 1676325482196Disabling compacts and flushes for region at 1676325482196Disabling writes for close at 1676325482197 (+1 ms)Writing region close event to WAL at 1676325482197Closed at 1676325482197 2023-02-13 21:58:02,230 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 21:58:02,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-13 21:58:02,599 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-1] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:58:02,599 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-1] regionserver.HRegion(1603): Closing ef75de132134b515dbe11abe569ac292, disabling compactions & flushes 2023-02-13 21:58:02,599 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-1] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. 2023-02-13 21:58:02,599 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. 2023-02-13 21:58:02,599 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-1] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. after waiting 0 ms 2023-02-13 21:58:02,599 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-1] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. 2023-02-13 21:58:02,599 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-1] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. 2023-02-13 21:58:02,599 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676325482126-pool-1] regionserver.HRegion(1557): Region close journal for ef75de132134b515dbe11abe569ac292: Waiting for close lock at 1676325482599Disabling compacts and flushes for region at 1676325482599Disabling writes for close at 1676325482599Writing region close event to WAL at 1676325482599Closed at 1676325482599 2023-02-13 21:58:02,601 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676325482126 execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 21:58:02,602 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676325482602"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325482602"}]},"ts":"1676325482602"} 2023-02-13 21:58:02,602 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325482602"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325482602"}]},"ts":"1676325482602"} 2023-02-13 21:58:02,602 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325482602"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325482602"}]},"ts":"1676325482602"} 2023-02-13 21:58:02,602 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676325482602"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325482602"}]},"ts":"1676325482602"} 2023-02-13 21:58:02,602 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325482602"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325482602"}]},"ts":"1676325482602"} 2023-02-13 21:58:02,602 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325482602"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325482602"}]},"ts":"1676325482602"} 2023-02-13 21:58:02,604 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-13 21:58:02,607 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676325482126 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 21:58:02,607 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676325482126","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325482607"}]},"ts":"1676325482607"} 2023-02-13 21:58:02,609 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676325482126, state=ENABLING in hbase:meta 2023-02-13 21:58:02,620 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-02-13 21:58:02,628 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 21:58:02,628 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 21:58:02,628 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 21:58:02,628 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 21:58:02,629 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=1c996d2a582a7ec43382d5a39bb54c78, ASSIGN}, {pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=f244b6e82e7a1752e9a3cb5f28dea2e9, ASSIGN}, {pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ac1d4ae96948f4de74aa3114b7308487, ASSIGN}, {pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=d1acdd49740dd716d5bec22786474a40, ASSIGN}, {pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ef75de132134b515dbe11abe569ac292, ASSIGN}, {pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=00d462c2fbf63cefa67cf8e8f681246f, ASSIGN}] 2023-02-13 21:58:02,632 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ac1d4ae96948f4de74aa3114b7308487, ASSIGN 2023-02-13 21:58:02,634 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=f244b6e82e7a1752e9a3cb5f28dea2e9, ASSIGN 2023-02-13 21:58:02,634 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=1c996d2a582a7ec43382d5a39bb54c78, ASSIGN 2023-02-13 21:58:02,634 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=d1acdd49740dd716d5bec22786474a40, ASSIGN 2023-02-13 21:58:02,635 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=f244b6e82e7a1752e9a3cb5f28dea2e9, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,45015,1676325451448; forceNewPlan=false, retain=false 2023-02-13 21:58:02,635 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=d1acdd49740dd716d5bec22786474a40, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,45015,1676325451448; forceNewPlan=false, retain=false 2023-02-13 21:58:02,635 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=1c996d2a582a7ec43382d5a39bb54c78, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,39523,1676325451480; forceNewPlan=false, retain=false 2023-02-13 21:58:02,635 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=00d462c2fbf63cefa67cf8e8f681246f, ASSIGN 2023-02-13 21:58:02,635 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ac1d4ae96948f4de74aa3114b7308487, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,36339,1676325451407; forceNewPlan=false, retain=false 2023-02-13 21:58:02,636 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ef75de132134b515dbe11abe569ac292, ASSIGN 2023-02-13 21:58:02,636 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=00d462c2fbf63cefa67cf8e8f681246f, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,39523,1676325451480; forceNewPlan=false, retain=false 2023-02-13 21:58:02,636 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ef75de132134b515dbe11abe569ac292, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,36339,1676325451407; forceNewPlan=false, retain=false 2023-02-13 21:58:02,785 INFO [jenkins-hbase20:46557] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 21:58:02,790 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=d1acdd49740dd716d5bec22786474a40, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:58:02,790 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=ac1d4ae96948f4de74aa3114b7308487, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:58:02,790 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=f244b6e82e7a1752e9a3cb5f28dea2e9, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:58:02,790 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325482790"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325482790"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325482790"}]},"ts":"1676325482790"} 2023-02-13 21:58:02,790 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=00d462c2fbf63cefa67cf8e8f681246f, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:58:02,790 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=1c996d2a582a7ec43382d5a39bb54c78, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:58:02,790 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676325482790"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325482790"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325482790"}]},"ts":"1676325482790"} 2023-02-13 21:58:02,790 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676325482790"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325482790"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325482790"}]},"ts":"1676325482790"} 2023-02-13 21:58:02,790 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325482790"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325482790"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325482790"}]},"ts":"1676325482790"} 2023-02-13 21:58:02,790 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325482790"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325482790"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325482790"}]},"ts":"1676325482790"} 2023-02-13 21:58:02,798 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=209, ppid=204, state=RUNNABLE; OpenRegionProcedure f244b6e82e7a1752e9a3cb5f28dea2e9, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:58:02,803 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=210, ppid=208, state=RUNNABLE; OpenRegionProcedure 00d462c2fbf63cefa67cf8e8f681246f, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:58:02,804 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=211, ppid=203, state=RUNNABLE; OpenRegionProcedure 1c996d2a582a7ec43382d5a39bb54c78, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:58:02,805 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=ef75de132134b515dbe11abe569ac292, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:58:02,805 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325482805"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325482805"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325482805"}]},"ts":"1676325482805"} 2023-02-13 21:58:02,805 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=212, ppid=205, state=RUNNABLE; OpenRegionProcedure ac1d4ae96948f4de74aa3114b7308487, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:58:02,807 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=213, ppid=206, state=RUNNABLE; OpenRegionProcedure d1acdd49740dd716d5bec22786474a40, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:58:02,809 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=214, ppid=207, state=RUNNABLE; OpenRegionProcedure ef75de132134b515dbe11abe569ac292, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:58:02,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-13 21:58:02,958 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. 2023-02-13 21:58:02,958 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 00d462c2fbf63cefa67cf8e8f681246f, NAME => 'testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 21:58:02,959 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676325482126 00d462c2fbf63cefa67cf8e8f681246f 2023-02-13 21:58:02,959 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:58:02,959 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 00d462c2fbf63cefa67cf8e8f681246f 2023-02-13 21:58:02,959 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. 2023-02-13 21:58:02,959 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f244b6e82e7a1752e9a3cb5f28dea2e9, NAME => 'testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 21:58:02,959 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 00d462c2fbf63cefa67cf8e8f681246f 2023-02-13 21:58:02,959 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676325482126 f244b6e82e7a1752e9a3cb5f28dea2e9 2023-02-13 21:58:02,959 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:58:02,959 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for f244b6e82e7a1752e9a3cb5f28dea2e9 2023-02-13 21:58:02,959 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for f244b6e82e7a1752e9a3cb5f28dea2e9 2023-02-13 21:58:02,960 INFO [StoreOpener-00d462c2fbf63cefa67cf8e8f681246f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 00d462c2fbf63cefa67cf8e8f681246f 2023-02-13 21:58:02,960 INFO [StoreOpener-f244b6e82e7a1752e9a3cb5f28dea2e9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f244b6e82e7a1752e9a3cb5f28dea2e9 2023-02-13 21:58:02,961 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. 2023-02-13 21:58:02,961 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ef75de132134b515dbe11abe569ac292, NAME => 'testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 21:58:02,961 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676325482126 ef75de132134b515dbe11abe569ac292 2023-02-13 21:58:02,962 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:58:02,962 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for ef75de132134b515dbe11abe569ac292 2023-02-13 21:58:02,962 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for ef75de132134b515dbe11abe569ac292 2023-02-13 21:58:02,962 DEBUG [StoreOpener-f244b6e82e7a1752e9a3cb5f28dea2e9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9/cf 2023-02-13 21:58:02,962 DEBUG [StoreOpener-f244b6e82e7a1752e9a3cb5f28dea2e9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9/cf 2023-02-13 21:58:02,963 DEBUG [StoreOpener-00d462c2fbf63cefa67cf8e8f681246f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/cf 2023-02-13 21:58:02,963 INFO [StoreOpener-ef75de132134b515dbe11abe569ac292-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ef75de132134b515dbe11abe569ac292 2023-02-13 21:58:02,963 DEBUG [StoreOpener-00d462c2fbf63cefa67cf8e8f681246f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/cf 2023-02-13 21:58:02,963 INFO [StoreOpener-f244b6e82e7a1752e9a3cb5f28dea2e9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f244b6e82e7a1752e9a3cb5f28dea2e9 columnFamilyName cf 2023-02-13 21:58:02,963 INFO [StoreOpener-00d462c2fbf63cefa67cf8e8f681246f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 00d462c2fbf63cefa67cf8e8f681246f columnFamilyName cf 2023-02-13 21:58:02,964 DEBUG [StoreOpener-ef75de132134b515dbe11abe569ac292-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292/cf 2023-02-13 21:58:02,964 DEBUG [StoreOpener-ef75de132134b515dbe11abe569ac292-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292/cf 2023-02-13 21:58:02,964 INFO [StoreOpener-f244b6e82e7a1752e9a3cb5f28dea2e9-1] regionserver.HStore(310): Store=f244b6e82e7a1752e9a3cb5f28dea2e9/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:58:02,965 INFO [StoreOpener-ef75de132134b515dbe11abe569ac292-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ef75de132134b515dbe11abe569ac292 columnFamilyName cf 2023-02-13 21:58:02,965 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9 2023-02-13 21:58:02,965 INFO [StoreOpener-ef75de132134b515dbe11abe569ac292-1] regionserver.HStore(310): Store=ef75de132134b515dbe11abe569ac292/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:58:02,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9 2023-02-13 21:58:02,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292 2023-02-13 21:58:02,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292 2023-02-13 21:58:02,967 INFO [StoreOpener-00d462c2fbf63cefa67cf8e8f681246f-1] regionserver.HStore(310): Store=00d462c2fbf63cefa67cf8e8f681246f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:58:02,967 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f 2023-02-13 21:58:02,968 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for f244b6e82e7a1752e9a3cb5f28dea2e9 2023-02-13 21:58:02,968 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f 2023-02-13 21:58:02,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for ef75de132134b515dbe11abe569ac292 2023-02-13 21:58:02,970 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 00d462c2fbf63cefa67cf8e8f681246f 2023-02-13 21:58:02,973 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 21:58:02,974 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 21:58:02,974 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened ef75de132134b515dbe11abe569ac292; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71477831, jitterRate=0.06510268151760101}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:58:02,974 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 00d462c2fbf63cefa67cf8e8f681246f; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58921005, jitterRate=-0.12200860679149628}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:58:02,974 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for ef75de132134b515dbe11abe569ac292: Running coprocessor pre-open hook at 1676325482962Writing region info on filesystem at 1676325482962Initializing all the Stores at 1676325482962Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325482962Cleaning up temporary data from old regions at 1676325482967 (+5 ms)Cleaning up detritus from prior splits at 1676325482967Running coprocessor post-open hooks at 1676325482974 (+7 ms)Region opened successfully at 1676325482974 2023-02-13 21:58:02,974 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 00d462c2fbf63cefa67cf8e8f681246f: Running coprocessor pre-open hook at 1676325482959Writing region info on filesystem at 1676325482959Initializing all the Stores at 1676325482960 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325482960Cleaning up temporary data from old regions at 1676325482968 (+8 ms)Cleaning up detritus from prior splits at 1676325482968Running coprocessor post-open hooks at 1676325482974 (+6 ms)Region opened successfully at 1676325482974 2023-02-13 21:58:02,974 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 21:58:02,975 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened f244b6e82e7a1752e9a3cb5f28dea2e9; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73116504, jitterRate=0.08952081203460693}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:58:02,975 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for f244b6e82e7a1752e9a3cb5f28dea2e9: Running coprocessor pre-open hook at 1676325482960Writing region info on filesystem at 1676325482960Initializing all the Stores at 1676325482960Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325482960Cleaning up temporary data from old regions at 1676325482966 (+6 ms)Cleaning up detritus from prior splits at 1676325482966Running coprocessor post-open hooks at 1676325482975 (+9 ms)Region opened successfully at 1676325482975 2023-02-13 21:58:02,975 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f., pid=210, masterSystemTime=1676325482955 2023-02-13 21:58:02,975 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292., pid=214, masterSystemTime=1676325482958 2023-02-13 21:58:02,975 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9., pid=209, masterSystemTime=1676325482953 2023-02-13 21:58:02,976 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. 2023-02-13 21:58:02,976 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. 2023-02-13 21:58:02,976 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=ef75de132134b515dbe11abe569ac292, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:58:02,976 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. 2023-02-13 21:58:02,977 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325482976"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325482976"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325482976"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325482976"}]},"ts":"1676325482976"} 2023-02-13 21:58:02,977 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ac1d4ae96948f4de74aa3114b7308487, NAME => 'testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 21:58:02,977 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676325482126 ac1d4ae96948f4de74aa3114b7308487 2023-02-13 21:58:02,977 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:58:02,977 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for ac1d4ae96948f4de74aa3114b7308487 2023-02-13 21:58:02,977 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=00d462c2fbf63cefa67cf8e8f681246f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:58:02,977 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for ac1d4ae96948f4de74aa3114b7308487 2023-02-13 21:58:02,977 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. 2023-02-13 21:58:02,977 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676325482977"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325482977"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325482977"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325482977"}]},"ts":"1676325482977"} 2023-02-13 21:58:02,978 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. 2023-02-13 21:58:02,978 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. 2023-02-13 21:58:02,978 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. 2023-02-13 21:58:02,978 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d1acdd49740dd716d5bec22786474a40, NAME => 'testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 21:58:02,978 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=f244b6e82e7a1752e9a3cb5f28dea2e9, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:58:02,978 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676325482126 d1acdd49740dd716d5bec22786474a40 2023-02-13 21:58:02,978 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:58:02,978 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for d1acdd49740dd716d5bec22786474a40 2023-02-13 21:58:02,978 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for d1acdd49740dd716d5bec22786474a40 2023-02-13 21:58:02,977 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. 2023-02-13 21:58:02,979 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. 2023-02-13 21:58:02,978 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325482978"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325482978"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325482978"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325482978"}]},"ts":"1676325482978"} 2023-02-13 21:58:02,979 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1c996d2a582a7ec43382d5a39bb54c78, NAME => 'testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 21:58:02,979 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676325482126 1c996d2a582a7ec43382d5a39bb54c78 2023-02-13 21:58:02,979 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:58:02,979 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 1c996d2a582a7ec43382d5a39bb54c78 2023-02-13 21:58:02,979 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 1c996d2a582a7ec43382d5a39bb54c78 2023-02-13 21:58:02,980 INFO [StoreOpener-ac1d4ae96948f4de74aa3114b7308487-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ac1d4ae96948f4de74aa3114b7308487 2023-02-13 21:58:02,982 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=214, resume processing ppid=207 2023-02-13 21:58:02,982 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=214, ppid=207, state=SUCCESS; OpenRegionProcedure ef75de132134b515dbe11abe569ac292, server=jenkins-hbase20.apache.org,36339,1676325451407 in 170 msec 2023-02-13 21:58:02,982 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=210, resume processing ppid=208 2023-02-13 21:58:02,982 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=210, ppid=208, state=SUCCESS; OpenRegionProcedure 00d462c2fbf63cefa67cf8e8f681246f, server=jenkins-hbase20.apache.org,39523,1676325451480 in 177 msec 2023-02-13 21:58:02,983 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=209, resume processing ppid=204 2023-02-13 21:58:02,983 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=209, ppid=204, state=SUCCESS; OpenRegionProcedure f244b6e82e7a1752e9a3cb5f28dea2e9, server=jenkins-hbase20.apache.org,45015,1676325451448 in 183 msec 2023-02-13 21:58:02,984 INFO [StoreOpener-1c996d2a582a7ec43382d5a39bb54c78-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1c996d2a582a7ec43382d5a39bb54c78 2023-02-13 21:58:02,984 DEBUG [StoreOpener-ac1d4ae96948f4de74aa3114b7308487-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487/cf 2023-02-13 21:58:02,984 DEBUG [StoreOpener-ac1d4ae96948f4de74aa3114b7308487-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487/cf 2023-02-13 21:58:02,984 INFO [StoreOpener-ac1d4ae96948f4de74aa3114b7308487-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ac1d4ae96948f4de74aa3114b7308487 columnFamilyName cf 2023-02-13 21:58:02,984 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=207, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ef75de132134b515dbe11abe569ac292, ASSIGN in 353 msec 2023-02-13 21:58:02,985 DEBUG [StoreOpener-1c996d2a582a7ec43382d5a39bb54c78-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78/cf 2023-02-13 21:58:02,985 DEBUG [StoreOpener-1c996d2a582a7ec43382d5a39bb54c78-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78/cf 2023-02-13 21:58:02,985 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=208, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=00d462c2fbf63cefa67cf8e8f681246f, ASSIGN in 353 msec 2023-02-13 21:58:02,985 INFO [StoreOpener-1c996d2a582a7ec43382d5a39bb54c78-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1c996d2a582a7ec43382d5a39bb54c78 columnFamilyName cf 2023-02-13 21:58:02,985 INFO [StoreOpener-d1acdd49740dd716d5bec22786474a40-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d1acdd49740dd716d5bec22786474a40 2023-02-13 21:58:02,985 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=204, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=f244b6e82e7a1752e9a3cb5f28dea2e9, ASSIGN in 354 msec 2023-02-13 21:58:02,985 INFO [StoreOpener-1c996d2a582a7ec43382d5a39bb54c78-1] regionserver.HStore(310): Store=1c996d2a582a7ec43382d5a39bb54c78/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:58:02,986 INFO [StoreOpener-ac1d4ae96948f4de74aa3114b7308487-1] regionserver.HStore(310): Store=ac1d4ae96948f4de74aa3114b7308487/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:58:02,986 DEBUG [StoreOpener-d1acdd49740dd716d5bec22786474a40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40/cf 2023-02-13 21:58:02,986 DEBUG [StoreOpener-d1acdd49740dd716d5bec22786474a40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40/cf 2023-02-13 21:58:02,986 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78 2023-02-13 21:58:02,986 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78 2023-02-13 21:58:02,986 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487 2023-02-13 21:58:02,986 INFO [StoreOpener-d1acdd49740dd716d5bec22786474a40-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d1acdd49740dd716d5bec22786474a40 columnFamilyName cf 2023-02-13 21:58:02,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487 2023-02-13 21:58:02,987 INFO [StoreOpener-d1acdd49740dd716d5bec22786474a40-1] regionserver.HStore(310): Store=d1acdd49740dd716d5bec22786474a40/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:58:02,988 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40 2023-02-13 21:58:02,988 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40 2023-02-13 21:58:02,988 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 1c996d2a582a7ec43382d5a39bb54c78 2023-02-13 21:58:02,988 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for ac1d4ae96948f4de74aa3114b7308487 2023-02-13 21:58:02,990 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for d1acdd49740dd716d5bec22786474a40 2023-02-13 21:58:02,991 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 21:58:02,991 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 21:58:02,991 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened ac1d4ae96948f4de74aa3114b7308487; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60513228, jitterRate=-0.09828263521194458}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:58:02,991 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for ac1d4ae96948f4de74aa3114b7308487: Running coprocessor pre-open hook at 1676325482977Writing region info on filesystem at 1676325482977Initializing all the Stores at 1676325482979 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325482979Cleaning up temporary data from old regions at 1676325482987 (+8 ms)Cleaning up detritus from prior splits at 1676325482987Running coprocessor post-open hooks at 1676325482991 (+4 ms)Region opened successfully at 1676325482991 2023-02-13 21:58:02,991 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 21:58:02,991 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 1c996d2a582a7ec43382d5a39bb54c78; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58886389, jitterRate=-0.12252442538738251}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:58:02,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 1c996d2a582a7ec43382d5a39bb54c78: Running coprocessor pre-open hook at 1676325482979Writing region info on filesystem at 1676325482979Initializing all the Stores at 1676325482983 (+4 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325482983Cleaning up temporary data from old regions at 1676325482987 (+4 ms)Cleaning up detritus from prior splits at 1676325482987Running coprocessor post-open hooks at 1676325482992 (+5 ms)Region opened successfully at 1676325482992 2023-02-13 21:58:02,992 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487., pid=212, masterSystemTime=1676325482958 2023-02-13 21:58:02,992 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78., pid=211, masterSystemTime=1676325482955 2023-02-13 21:58:02,992 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened d1acdd49740dd716d5bec22786474a40; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58920605, jitterRate=-0.12201456725597382}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:58:02,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for d1acdd49740dd716d5bec22786474a40: Running coprocessor pre-open hook at 1676325482978Writing region info on filesystem at 1676325482978Initializing all the Stores at 1676325482984 (+6 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325482984Cleaning up temporary data from old regions at 1676325482988 (+4 ms)Cleaning up detritus from prior splits at 1676325482989 (+1 ms)Running coprocessor post-open hooks at 1676325482992 (+3 ms)Region opened successfully at 1676325482992 2023-02-13 21:58:02,993 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40., pid=213, masterSystemTime=1676325482953 2023-02-13 21:58:02,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. 2023-02-13 21:58:02,994 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. 2023-02-13 21:58:02,994 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=ac1d4ae96948f4de74aa3114b7308487, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:58:02,994 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325482994"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325482994"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325482994"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325482994"}]},"ts":"1676325482994"} 2023-02-13 21:58:02,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. 2023-02-13 21:58:02,994 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. 2023-02-13 21:58:02,995 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=1c996d2a582a7ec43382d5a39bb54c78, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:58:02,995 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. 2023-02-13 21:58:02,995 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676325482995"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325482995"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325482995"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325482995"}]},"ts":"1676325482995"} 2023-02-13 21:58:02,995 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. 2023-02-13 21:58:02,995 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=d1acdd49740dd716d5bec22786474a40, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:58:02,995 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325482995"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325482995"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325482995"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325482995"}]},"ts":"1676325482995"} 2023-02-13 21:58:02,997 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=212, resume processing ppid=205 2023-02-13 21:58:02,997 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=212, ppid=205, state=SUCCESS; OpenRegionProcedure ac1d4ae96948f4de74aa3114b7308487, server=jenkins-hbase20.apache.org,36339,1676325451407 in 190 msec 2023-02-13 21:58:02,997 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=211, resume processing ppid=203 2023-02-13 21:58:02,997 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=211, ppid=203, state=SUCCESS; OpenRegionProcedure 1c996d2a582a7ec43382d5a39bb54c78, server=jenkins-hbase20.apache.org,39523,1676325451480 in 192 msec 2023-02-13 21:58:02,998 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=213, resume processing ppid=206 2023-02-13 21:58:02,998 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=213, ppid=206, state=SUCCESS; OpenRegionProcedure d1acdd49740dd716d5bec22786474a40, server=jenkins-hbase20.apache.org,45015,1676325451448 in 189 msec 2023-02-13 21:58:02,998 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=205, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ac1d4ae96948f4de74aa3114b7308487, ASSIGN in 368 msec 2023-02-13 21:58:02,998 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=203, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=1c996d2a582a7ec43382d5a39bb54c78, ASSIGN in 369 msec 2023-02-13 21:58:02,999 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=206, resume processing ppid=202 2023-02-13 21:58:02,999 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=206, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=d1acdd49740dd716d5bec22786474a40, ASSIGN in 369 msec 2023-02-13 21:58:03,000 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676325482126 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 21:58:03,000 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676325482126","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325483000"}]},"ts":"1676325483000"} 2023-02-13 21:58:03,001 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676325482126, state=ENABLED in hbase:meta 2023-02-13 21:58:03,007 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676325482126 execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 21:58:03,012 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=202, state=SUCCESS; CreateTableProcedure table=testCorruptedSnapshot-1676325482126 in 880 msec 2023-02-13 21:58:03,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-13 21:58:03,637 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testCorruptedSnapshot-1676325482126, procId: 202 completed 2023-02-13 21:58:03,649 DEBUG [Listener at localhost.localdomain/44075] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676325482126 2023-02-13 21:58:03,649 DEBUG [Listener at localhost.localdomain/44075] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. 2023-02-13 21:58:03,654 INFO [Listener at localhost.localdomain/44075] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 21:58:03,678 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676325482126 2023-02-13 21:58:03,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.HMaster$11(2405): Client=jenkins//148.251.75.209 disable testCorruptedSnapshot-1676325482126 2023-02-13 21:58:03,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=215, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676325482126 2023-02-13 21:58:03,692 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676325482126","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325483692"}]},"ts":"1676325483692"} 2023-02-13 21:58:03,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-13 21:58:03,694 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676325482126, state=DISABLING in hbase:meta 2023-02-13 21:58:03,695 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676325482126 to state=DISABLING 2023-02-13 21:58:03,697 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=1c996d2a582a7ec43382d5a39bb54c78, UNASSIGN}, {pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=f244b6e82e7a1752e9a3cb5f28dea2e9, UNASSIGN}, {pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ac1d4ae96948f4de74aa3114b7308487, UNASSIGN}, {pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=d1acdd49740dd716d5bec22786474a40, UNASSIGN}, {pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ef75de132134b515dbe11abe569ac292, UNASSIGN}, {pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=00d462c2fbf63cefa67cf8e8f681246f, UNASSIGN}] 2023-02-13 21:58:03,699 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ef75de132134b515dbe11abe569ac292, UNASSIGN 2023-02-13 21:58:03,700 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=00d462c2fbf63cefa67cf8e8f681246f, UNASSIGN 2023-02-13 21:58:03,700 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=d1acdd49740dd716d5bec22786474a40, UNASSIGN 2023-02-13 21:58:03,700 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ac1d4ae96948f4de74aa3114b7308487, UNASSIGN 2023-02-13 21:58:03,701 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=f244b6e82e7a1752e9a3cb5f28dea2e9, UNASSIGN 2023-02-13 21:58:03,701 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=ef75de132134b515dbe11abe569ac292, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:58:03,701 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325483701"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325483701"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325483701"}]},"ts":"1676325483701"} 2023-02-13 21:58:03,703 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=222, ppid=220, state=RUNNABLE; CloseRegionProcedure ef75de132134b515dbe11abe569ac292, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:58:03,715 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=00d462c2fbf63cefa67cf8e8f681246f, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:58:03,715 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=d1acdd49740dd716d5bec22786474a40, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:58:03,715 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676325483715"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325483715"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325483715"}]},"ts":"1676325483715"} 2023-02-13 21:58:03,715 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325483715"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325483715"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325483715"}]},"ts":"1676325483715"} 2023-02-13 21:58:03,716 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=ac1d4ae96948f4de74aa3114b7308487, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:58:03,716 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325483716"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325483716"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325483716"}]},"ts":"1676325483716"} 2023-02-13 21:58:03,716 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=f244b6e82e7a1752e9a3cb5f28dea2e9, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:58:03,717 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325483716"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325483716"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325483716"}]},"ts":"1676325483716"} 2023-02-13 21:58:03,718 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=223, ppid=219, state=RUNNABLE; CloseRegionProcedure d1acdd49740dd716d5bec22786474a40, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:58:03,720 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=224, ppid=221, state=RUNNABLE; CloseRegionProcedure 00d462c2fbf63cefa67cf8e8f681246f, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:58:03,720 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=1c996d2a582a7ec43382d5a39bb54c78, UNASSIGN 2023-02-13 21:58:03,721 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=225, ppid=218, state=RUNNABLE; CloseRegionProcedure ac1d4ae96948f4de74aa3114b7308487, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:58:03,722 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=226, ppid=217, state=RUNNABLE; CloseRegionProcedure f244b6e82e7a1752e9a3cb5f28dea2e9, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:58:03,722 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=1c996d2a582a7ec43382d5a39bb54c78, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:58:03,723 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676325483722"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325483722"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325483722"}]},"ts":"1676325483722"} 2023-02-13 21:58:03,724 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=227, ppid=216, state=RUNNABLE; CloseRegionProcedure 1c996d2a582a7ec43382d5a39bb54c78, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:58:03,869 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close ef75de132134b515dbe11abe569ac292 2023-02-13 21:58:03,869 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing ef75de132134b515dbe11abe569ac292, disabling compactions & flushes 2023-02-13 21:58:03,869 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. 2023-02-13 21:58:03,869 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. 2023-02-13 21:58:03,869 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. after waiting 0 ms 2023-02-13 21:58:03,869 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. 2023-02-13 21:58:03,874 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 00d462c2fbf63cefa67cf8e8f681246f 2023-02-13 21:58:03,874 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close d1acdd49740dd716d5bec22786474a40 2023-02-13 21:58:03,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing d1acdd49740dd716d5bec22786474a40, disabling compactions & flushes 2023-02-13 21:58:03,874 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. 2023-02-13 21:58:03,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. 2023-02-13 21:58:03,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. after waiting 0 ms 2023-02-13 21:58:03,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. 2023-02-13 21:58:03,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 00d462c2fbf63cefa67cf8e8f681246f, disabling compactions & flushes 2023-02-13 21:58:03,875 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. 2023-02-13 21:58:03,875 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. 2023-02-13 21:58:03,875 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. after waiting 0 ms 2023-02-13 21:58:03,875 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. 2023-02-13 21:58:03,927 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 21:58:03,927 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 21:58:03,928 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 21:58:03,928 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. 2023-02-13 21:58:03,928 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for d1acdd49740dd716d5bec22786474a40: Waiting for close lock at 1676325483874Running coprocessor pre-close hooks at 1676325483874Disabling compacts and flushes for region at 1676325483874Disabling writes for close at 1676325483874Writing region close event to WAL at 1676325483923 (+49 ms)Running coprocessor post-close hooks at 1676325483928 (+5 ms)Closed at 1676325483928 2023-02-13 21:58:03,928 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. 2023-02-13 21:58:03,928 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 00d462c2fbf63cefa67cf8e8f681246f: Waiting for close lock at 1676325483874Running coprocessor pre-close hooks at 1676325483874Disabling compacts and flushes for region at 1676325483874Disabling writes for close at 1676325483875 (+1 ms)Writing region close event to WAL at 1676325483923 (+48 ms)Running coprocessor post-close hooks at 1676325483928 (+5 ms)Closed at 1676325483928 2023-02-13 21:58:03,929 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. 2023-02-13 21:58:03,929 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for ef75de132134b515dbe11abe569ac292: Waiting for close lock at 1676325483869Running coprocessor pre-close hooks at 1676325483869Disabling compacts and flushes for region at 1676325483869Disabling writes for close at 1676325483869Writing region close event to WAL at 1676325483923 (+54 ms)Running coprocessor post-close hooks at 1676325483929 (+6 ms)Closed at 1676325483929 2023-02-13 21:58:03,932 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed d1acdd49740dd716d5bec22786474a40 2023-02-13 21:58:03,932 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close f244b6e82e7a1752e9a3cb5f28dea2e9 2023-02-13 21:58:03,932 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing f244b6e82e7a1752e9a3cb5f28dea2e9, disabling compactions & flushes 2023-02-13 21:58:03,932 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. 2023-02-13 21:58:03,932 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. 2023-02-13 21:58:03,932 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. after waiting 0 ms 2023-02-13 21:58:03,932 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. 2023-02-13 21:58:03,935 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=d1acdd49740dd716d5bec22786474a40, regionState=CLOSED 2023-02-13 21:58:03,935 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325483935"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325483935"}]},"ts":"1676325483935"} 2023-02-13 21:58:03,935 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 00d462c2fbf63cefa67cf8e8f681246f 2023-02-13 21:58:03,935 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 1c996d2a582a7ec43382d5a39bb54c78 2023-02-13 21:58:03,935 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 1c996d2a582a7ec43382d5a39bb54c78, disabling compactions & flushes 2023-02-13 21:58:03,935 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. 2023-02-13 21:58:03,935 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. 2023-02-13 21:58:03,935 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. after waiting 0 ms 2023-02-13 21:58:03,935 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. 2023-02-13 21:58:03,937 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed ef75de132134b515dbe11abe569ac292 2023-02-13 21:58:03,937 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close ac1d4ae96948f4de74aa3114b7308487 2023-02-13 21:58:03,938 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing ac1d4ae96948f4de74aa3114b7308487, disabling compactions & flushes 2023-02-13 21:58:03,938 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. 2023-02-13 21:58:03,938 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. 2023-02-13 21:58:03,938 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. after waiting 0 ms 2023-02-13 21:58:03,938 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. 2023-02-13 21:58:03,943 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=223, resume processing ppid=219 2023-02-13 21:58:03,943 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=223, ppid=219, state=SUCCESS; CloseRegionProcedure d1acdd49740dd716d5bec22786474a40, server=jenkins-hbase20.apache.org,45015,1676325451448 in 220 msec 2023-02-13 21:58:03,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-13 21:58:03,944 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=ef75de132134b515dbe11abe569ac292, regionState=CLOSED 2023-02-13 21:58:03,944 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=00d462c2fbf63cefa67cf8e8f681246f, regionState=CLOSED 2023-02-13 21:58:03,945 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325483944"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325483944"}]},"ts":"1676325483944"} 2023-02-13 21:58:03,945 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676325483944"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325483944"}]},"ts":"1676325483944"} 2023-02-13 21:58:03,948 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=219, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=d1acdd49740dd716d5bec22786474a40, UNASSIGN in 247 msec 2023-02-13 21:58:03,957 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=224, resume processing ppid=221 2023-02-13 21:58:03,957 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=224, ppid=221, state=SUCCESS; CloseRegionProcedure 00d462c2fbf63cefa67cf8e8f681246f, server=jenkins-hbase20.apache.org,39523,1676325451480 in 227 msec 2023-02-13 21:58:03,958 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=222, resume processing ppid=220 2023-02-13 21:58:03,958 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=222, ppid=220, state=SUCCESS; CloseRegionProcedure ef75de132134b515dbe11abe569ac292, server=jenkins-hbase20.apache.org,36339,1676325451407 in 246 msec 2023-02-13 21:58:03,960 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=221, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=00d462c2fbf63cefa67cf8e8f681246f, UNASSIGN in 260 msec 2023-02-13 21:58:03,961 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=220, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ef75de132134b515dbe11abe569ac292, UNASSIGN in 261 msec 2023-02-13 21:58:03,987 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 21:58:03,988 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. 2023-02-13 21:58:03,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 1c996d2a582a7ec43382d5a39bb54c78: Waiting for close lock at 1676325483935Running coprocessor pre-close hooks at 1676325483935Disabling compacts and flushes for region at 1676325483935Disabling writes for close at 1676325483935Writing region close event to WAL at 1676325483946 (+11 ms)Running coprocessor post-close hooks at 1676325483987 (+41 ms)Closed at 1676325483988 (+1 ms) 2023-02-13 21:58:03,992 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 1c996d2a582a7ec43382d5a39bb54c78 2023-02-13 21:58:03,993 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=1c996d2a582a7ec43382d5a39bb54c78, regionState=CLOSED 2023-02-13 21:58:03,993 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676325483993"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325483993"}]},"ts":"1676325483993"} 2023-02-13 21:58:03,996 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 21:58:04,005 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 21:58:04,005 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. 2023-02-13 21:58:04,005 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for ac1d4ae96948f4de74aa3114b7308487: Waiting for close lock at 1676325483938Running coprocessor pre-close hooks at 1676325483938Disabling compacts and flushes for region at 1676325483938Disabling writes for close at 1676325483938Writing region close event to WAL at 1676325483946 (+8 ms)Running coprocessor post-close hooks at 1676325484005 (+59 ms)Closed at 1676325484005 2023-02-13 21:58:04,005 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. 2023-02-13 21:58:04,005 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for f244b6e82e7a1752e9a3cb5f28dea2e9: Waiting for close lock at 1676325483932Running coprocessor pre-close hooks at 1676325483932Disabling compacts and flushes for region at 1676325483932Disabling writes for close at 1676325483932Writing region close event to WAL at 1676325483946 (+14 ms)Running coprocessor post-close hooks at 1676325484005 (+59 ms)Closed at 1676325484005 2023-02-13 21:58:04,006 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=227, resume processing ppid=216 2023-02-13 21:58:04,006 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=227, ppid=216, state=SUCCESS; CloseRegionProcedure 1c996d2a582a7ec43382d5a39bb54c78, server=jenkins-hbase20.apache.org,39523,1676325451480 in 272 msec 2023-02-13 21:58:04,007 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed ac1d4ae96948f4de74aa3114b7308487 2023-02-13 21:58:04,009 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=216, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=1c996d2a582a7ec43382d5a39bb54c78, UNASSIGN in 309 msec 2023-02-13 21:58:04,009 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=ac1d4ae96948f4de74aa3114b7308487, regionState=CLOSED 2023-02-13 21:58:04,009 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325484009"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325484009"}]},"ts":"1676325484009"} 2023-02-13 21:58:04,010 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed f244b6e82e7a1752e9a3cb5f28dea2e9 2023-02-13 21:58:04,010 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=f244b6e82e7a1752e9a3cb5f28dea2e9, regionState=CLOSED 2023-02-13 21:58:04,010 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325484010"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325484010"}]},"ts":"1676325484010"} 2023-02-13 21:58:04,016 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=225, resume processing ppid=218 2023-02-13 21:58:04,016 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=225, ppid=218, state=SUCCESS; CloseRegionProcedure ac1d4ae96948f4de74aa3114b7308487, server=jenkins-hbase20.apache.org,36339,1676325451407 in 290 msec 2023-02-13 21:58:04,018 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=226, resume processing ppid=217 2023-02-13 21:58:04,018 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=226, ppid=217, state=SUCCESS; CloseRegionProcedure f244b6e82e7a1752e9a3cb5f28dea2e9, server=jenkins-hbase20.apache.org,45015,1676325451448 in 291 msec 2023-02-13 21:58:04,018 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=218, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ac1d4ae96948f4de74aa3114b7308487, UNASSIGN in 319 msec 2023-02-13 21:58:04,024 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=217, resume processing ppid=215 2023-02-13 21:58:04,024 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=217, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=f244b6e82e7a1752e9a3cb5f28dea2e9, UNASSIGN in 321 msec 2023-02-13 21:58:04,025 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676325482126","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325484025"}]},"ts":"1676325484025"} 2023-02-13 21:58:04,027 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676325482126, state=DISABLED in hbase:meta 2023-02-13 21:58:04,028 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676325482126 to state=DISABLED 2023-02-13 21:58:04,031 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=215, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676325482126 in 348 msec 2023-02-13 21:58:04,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-13 21:58:04,446 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676325482126, procId: 215 completed 2023-02-13 21:58:04,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1580): Client=jenkins//148.251.75.209 snapshot request for:{ ss=emptySnaptb-1676325482126 table=testCorruptedSnapshot-1676325482126 type=FLUSH ttl=0 } 2023-02-13 21:58:04,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676325484447 (current time:1676325484447). 2023-02-13 21:58:04,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 21:58:04,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] zookeeper.ReadOnlyZKClient(139): Connect 0x180a6f56 to 127.0.0.1:58580 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 21:58:04,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f7d3b3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 21:58:04,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 21:58:04,494 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60256, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 21:58:04,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x180a6f56 to 127.0.0.1:58580 2023-02-13 21:58:04,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 21:58:04,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 21:58:04,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1676325482126 table=testCorruptedSnapshot-1676325482126 type=FLUSH ttl=0 } 2023-02-13 21:58:04,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676325482126, type=EXCLUSIVE 2023-02-13 21:58:04,514 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676325482126, type=EXCLUSIVE 2023-02-13 21:58:04,515 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=228, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676325482126, type=EXCLUSIVE; timeout=600000, timestamp=1676326084515 2023-02-13 21:58:04,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1676325482126 table=testCorruptedSnapshot-1676325482126 type=FLUSH ttl=0 } 2023-02-13 21:58:04,516 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1676325482126 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676325482126 2023-02-13 21:58:04,517 DEBUG [Listener at localhost.localdomain/44075] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1676325482126 table=testCorruptedSnapshot-1676325482126 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 21:58:04,517 DEBUG [Listener at localhost.localdomain/44075] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 21:58:04,564 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676325482126 table=testCorruptedSnapshot-1676325482126 type=DISABLED ttl=0 } 2023-02-13 21:58:04,596 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:58:04,596 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:58:04,596 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:58:04,596 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:58:04,596 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:58:04,596 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:58:04,596 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:58:04,597 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:58:04,597 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:58:04,597 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:58:04,597 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:58:04,597 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:58:04,597 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:58:04,598 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:58:04,599 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 21:58:04,599 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 21:58:04,599 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 21:58:04,604 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 21:58:04,606 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 21:58:04,608 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 21:58:04,610 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 21:58:04,756 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1676325482126 table=testCorruptedSnapshot-1676325482126 type=DISABLED ttl=0 } as finished. 2023-02-13 21:58:04,757 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1676325482126 2023-02-13 21:58:04,757 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/emptySnaptb-1676325482126 2023-02-13 21:58:04,767 DEBUG [Listener at localhost.localdomain/44075] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 21:58:04,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676325482126 table=testCorruptedSnapshot-1676325482126 type=FLUSH ttl=0 } is done 2023-02-13 21:58:04,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(404): Snapshoting '{ ss=emptySnaptb-1676325482126 table=testCorruptedSnapshot-1676325482126 type=FLUSH ttl=0 }' is still in progress! 2023-02-13 21:58:04,768 DEBUG [Listener at localhost.localdomain/44075] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-13 21:58:04,783 WARN [DataStreamer for file /user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/emptySnaptb-1676325482126/data.manifest] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-02-13 21:58:04,803 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/emptySnaptb-1676325482126 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/emptySnaptb-1676325482126 2023-02-13 21:58:04,863 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1676325482126 of table testCorruptedSnapshot-1676325482126 completed 2023-02-13 21:58:04,863 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/emptySnaptb-1676325482126 2023-02-13 21:58:04,864 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/emptySnaptb-1676325482126 2023-02-13 21:58:04,864 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1676325482126 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676325482126 at 1676325484517Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676325482126 table=testCorruptedSnapshot-1676325482126 type=DISABLED ttl=0 } at 1676325484564 (+47 ms)Consolidate snapshot: emptySnaptb-1676325482126 at 1676325484756 (+192 ms)Loading Region manifests for emptySnaptb-1676325482126 at 1676325484757 (+1 ms)Writing data manifest for emptySnaptb-1676325482126 at 1676325484765 (+8 ms)Verifying snapshot: emptySnaptb-1676325482126 at 1676325484791 (+26 ms)Snapshot emptySnaptb-1676325482126 of table testCorruptedSnapshot-1676325482126 completed at 1676325484863 (+72 ms) 2023-02-13 21:58:04,866 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=228, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676325482126, type=EXCLUSIVE 2023-02-13 21:58:04,868 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=228, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676325482126, type=EXCLUSIVE in 354 msec 2023-02-13 21:58:05,269 DEBUG [Listener at localhost.localdomain/44075] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 21:58:05,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676325482126 table=testCorruptedSnapshot-1676325482126 type=FLUSH ttl=0 } is done 2023-02-13 21:58:05,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1676325482126 table=testCorruptedSnapshot-1676325482126 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 21:58:05,272 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1676325482126 2023-02-13 21:58:05,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.HMaster$10(2372): Client=jenkins//148.251.75.209 enable testCorruptedSnapshot-1676325482126 2023-02-13 21:58:05,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=229, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1676325482126 2023-02-13 21:58:05,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-13 21:58:05,277 INFO [PEWorker-5] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1676325482126 2023-02-13 21:58:05,277 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676325482126","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325485277"}]},"ts":"1676325485277"} 2023-02-13 21:58:05,279 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676325482126, state=ENABLING in hbase:meta 2023-02-13 21:58:05,281 DEBUG [PEWorker-5] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 21:58:05,281 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=1c996d2a582a7ec43382d5a39bb54c78, ASSIGN}, {pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=f244b6e82e7a1752e9a3cb5f28dea2e9, ASSIGN}, {pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ac1d4ae96948f4de74aa3114b7308487, ASSIGN}, {pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=d1acdd49740dd716d5bec22786474a40, ASSIGN}, {pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ef75de132134b515dbe11abe569ac292, ASSIGN}, {pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=00d462c2fbf63cefa67cf8e8f681246f, ASSIGN}] 2023-02-13 21:58:05,285 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ef75de132134b515dbe11abe569ac292, ASSIGN 2023-02-13 21:58:05,285 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=d1acdd49740dd716d5bec22786474a40, ASSIGN 2023-02-13 21:58:05,285 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ac1d4ae96948f4de74aa3114b7308487, ASSIGN 2023-02-13 21:58:05,285 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=f244b6e82e7a1752e9a3cb5f28dea2e9, ASSIGN 2023-02-13 21:58:05,286 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:58:05,286 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:58:05,286 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=d1acdd49740dd716d5bec22786474a40, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,45015,1676325451448; forceNewPlan=false, retain=true 2023-02-13 21:58:05,286 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=00d462c2fbf63cefa67cf8e8f681246f, ASSIGN 2023-02-13 21:58:05,286 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ac1d4ae96948f4de74aa3114b7308487, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,36339,1676325451407; forceNewPlan=false, retain=true 2023-02-13 21:58:05,286 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:58:05,287 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ef75de132134b515dbe11abe569ac292, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,36339,1676325451407; forceNewPlan=false, retain=true 2023-02-13 21:58:05,286 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:58:05,287 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=f244b6e82e7a1752e9a3cb5f28dea2e9, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,45015,1676325451448; forceNewPlan=false, retain=true 2023-02-13 21:58:05,289 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=1c996d2a582a7ec43382d5a39bb54c78, ASSIGN 2023-02-13 21:58:05,289 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:58:05,289 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=00d462c2fbf63cefa67cf8e8f681246f, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,39523,1676325451480; forceNewPlan=false, retain=true 2023-02-13 21:58:05,290 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:58:05,290 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=1c996d2a582a7ec43382d5a39bb54c78, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,39523,1676325451480; forceNewPlan=false, retain=true 2023-02-13 21:58:05,436 INFO [jenkins-hbase20:46557] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 21:58:05,437 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=d1acdd49740dd716d5bec22786474a40, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:58:05,437 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=00d462c2fbf63cefa67cf8e8f681246f, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:58:05,437 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325485437"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325485437"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325485437"}]},"ts":"1676325485437"} 2023-02-13 21:58:05,437 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676325485437"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325485437"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325485437"}]},"ts":"1676325485437"} 2023-02-13 21:58:05,437 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=1c996d2a582a7ec43382d5a39bb54c78, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:58:05,437 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676325485437"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325485437"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325485437"}]},"ts":"1676325485437"} 2023-02-13 21:58:05,437 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=f244b6e82e7a1752e9a3cb5f28dea2e9, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:58:05,437 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=ac1d4ae96948f4de74aa3114b7308487, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:58:05,438 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325485437"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325485437"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325485437"}]},"ts":"1676325485437"} 2023-02-13 21:58:05,438 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325485437"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325485437"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325485437"}]},"ts":"1676325485437"} 2023-02-13 21:58:05,439 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=236, ppid=233, state=RUNNABLE; OpenRegionProcedure d1acdd49740dd716d5bec22786474a40, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:58:05,446 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=237, ppid=235, state=RUNNABLE; OpenRegionProcedure 00d462c2fbf63cefa67cf8e8f681246f, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:58:05,448 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=238, ppid=230, state=RUNNABLE; OpenRegionProcedure 1c996d2a582a7ec43382d5a39bb54c78, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:58:05,451 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=239, ppid=232, state=RUNNABLE; OpenRegionProcedure ac1d4ae96948f4de74aa3114b7308487, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:58:05,452 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=ef75de132134b515dbe11abe569ac292, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:58:05,452 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325485452"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325485452"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325485452"}]},"ts":"1676325485452"} 2023-02-13 21:58:05,458 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=240, ppid=231, state=RUNNABLE; OpenRegionProcedure f244b6e82e7a1752e9a3cb5f28dea2e9, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:58:05,468 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=241, ppid=234, state=RUNNABLE; OpenRegionProcedure ef75de132134b515dbe11abe569ac292, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:58:05,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-13 21:58:05,604 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. 2023-02-13 21:58:05,604 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 00d462c2fbf63cefa67cf8e8f681246f, NAME => 'testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 21:58:05,604 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676325482126 00d462c2fbf63cefa67cf8e8f681246f 2023-02-13 21:58:05,604 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:58:05,604 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 00d462c2fbf63cefa67cf8e8f681246f 2023-02-13 21:58:05,604 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 00d462c2fbf63cefa67cf8e8f681246f 2023-02-13 21:58:05,605 INFO [StoreOpener-00d462c2fbf63cefa67cf8e8f681246f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 00d462c2fbf63cefa67cf8e8f681246f 2023-02-13 21:58:05,606 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. 2023-02-13 21:58:05,606 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f244b6e82e7a1752e9a3cb5f28dea2e9, NAME => 'testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 21:58:05,606 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676325482126 f244b6e82e7a1752e9a3cb5f28dea2e9 2023-02-13 21:58:05,606 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:58:05,606 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for f244b6e82e7a1752e9a3cb5f28dea2e9 2023-02-13 21:58:05,606 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for f244b6e82e7a1752e9a3cb5f28dea2e9 2023-02-13 21:58:05,618 DEBUG [StoreOpener-00d462c2fbf63cefa67cf8e8f681246f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/cf 2023-02-13 21:58:05,619 DEBUG [StoreOpener-00d462c2fbf63cefa67cf8e8f681246f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/cf 2023-02-13 21:58:05,619 INFO [StoreOpener-00d462c2fbf63cefa67cf8e8f681246f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 00d462c2fbf63cefa67cf8e8f681246f columnFamilyName cf 2023-02-13 21:58:05,619 INFO [StoreOpener-f244b6e82e7a1752e9a3cb5f28dea2e9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f244b6e82e7a1752e9a3cb5f28dea2e9 2023-02-13 21:58:05,620 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. 2023-02-13 21:58:05,620 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ef75de132134b515dbe11abe569ac292, NAME => 'testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 21:58:05,620 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676325482126 ef75de132134b515dbe11abe569ac292 2023-02-13 21:58:05,620 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:58:05,620 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for ef75de132134b515dbe11abe569ac292 2023-02-13 21:58:05,621 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for ef75de132134b515dbe11abe569ac292 2023-02-13 21:58:05,623 INFO [StoreOpener-00d462c2fbf63cefa67cf8e8f681246f-1] regionserver.HStore(310): Store=00d462c2fbf63cefa67cf8e8f681246f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:58:05,624 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f 2023-02-13 21:58:05,624 INFO [StoreOpener-ef75de132134b515dbe11abe569ac292-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ef75de132134b515dbe11abe569ac292 2023-02-13 21:58:05,624 DEBUG [StoreOpener-f244b6e82e7a1752e9a3cb5f28dea2e9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9/cf 2023-02-13 21:58:05,624 DEBUG [StoreOpener-f244b6e82e7a1752e9a3cb5f28dea2e9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9/cf 2023-02-13 21:58:05,625 INFO [StoreOpener-f244b6e82e7a1752e9a3cb5f28dea2e9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f244b6e82e7a1752e9a3cb5f28dea2e9 columnFamilyName cf 2023-02-13 21:58:05,625 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f 2023-02-13 21:58:05,626 DEBUG [StoreOpener-ef75de132134b515dbe11abe569ac292-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292/cf 2023-02-13 21:58:05,626 DEBUG [StoreOpener-ef75de132134b515dbe11abe569ac292-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292/cf 2023-02-13 21:58:05,626 INFO [StoreOpener-f244b6e82e7a1752e9a3cb5f28dea2e9-1] regionserver.HStore(310): Store=f244b6e82e7a1752e9a3cb5f28dea2e9/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:58:05,627 INFO [StoreOpener-ef75de132134b515dbe11abe569ac292-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ef75de132134b515dbe11abe569ac292 columnFamilyName cf 2023-02-13 21:58:05,627 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9 2023-02-13 21:58:05,629 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9 2023-02-13 21:58:05,629 INFO [StoreOpener-ef75de132134b515dbe11abe569ac292-1] regionserver.HStore(310): Store=ef75de132134b515dbe11abe569ac292/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:58:05,629 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 00d462c2fbf63cefa67cf8e8f681246f 2023-02-13 21:58:05,630 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292 2023-02-13 21:58:05,631 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 00d462c2fbf63cefa67cf8e8f681246f; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60612862, jitterRate=-0.09679797291755676}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:58:05,631 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 00d462c2fbf63cefa67cf8e8f681246f: Running coprocessor pre-open hook at 1676325485604Writing region info on filesystem at 1676325485604Initializing all the Stores at 1676325485605 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325485605Cleaning up temporary data from old regions at 1676325485626 (+21 ms)Cleaning up detritus from prior splits at 1676325485626Running coprocessor post-open hooks at 1676325485631 (+5 ms)Region opened successfully at 1676325485631 2023-02-13 21:58:05,633 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f., pid=237, masterSystemTime=1676325485601 2023-02-13 21:58:05,633 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292 2023-02-13 21:58:05,633 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for f244b6e82e7a1752e9a3cb5f28dea2e9 2023-02-13 21:58:05,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. 2023-02-13 21:58:05,635 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. 2023-02-13 21:58:05,635 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. 2023-02-13 21:58:05,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1c996d2a582a7ec43382d5a39bb54c78, NAME => 'testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 21:58:05,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676325482126 1c996d2a582a7ec43382d5a39bb54c78 2023-02-13 21:58:05,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:58:05,635 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened f244b6e82e7a1752e9a3cb5f28dea2e9; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65332325, jitterRate=-0.02647249400615692}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:58:05,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 1c996d2a582a7ec43382d5a39bb54c78 2023-02-13 21:58:05,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for f244b6e82e7a1752e9a3cb5f28dea2e9: Running coprocessor pre-open hook at 1676325485606Writing region info on filesystem at 1676325485606Initializing all the Stores at 1676325485618 (+12 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325485618Cleaning up temporary data from old regions at 1676325485629 (+11 ms)Cleaning up detritus from prior splits at 1676325485630 (+1 ms)Running coprocessor post-open hooks at 1676325485635 (+5 ms)Region opened successfully at 1676325485635 2023-02-13 21:58:05,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 1c996d2a582a7ec43382d5a39bb54c78 2023-02-13 21:58:05,637 INFO [StoreOpener-1c996d2a582a7ec43382d5a39bb54c78-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1c996d2a582a7ec43382d5a39bb54c78 2023-02-13 21:58:05,638 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=00d462c2fbf63cefa67cf8e8f681246f, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:58:05,638 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676325485638"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325485638"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325485638"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325485638"}]},"ts":"1676325485638"} 2023-02-13 21:58:05,639 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9., pid=240, masterSystemTime=1676325485596 2023-02-13 21:58:05,640 DEBUG [StoreOpener-1c996d2a582a7ec43382d5a39bb54c78-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78/cf 2023-02-13 21:58:05,640 DEBUG [StoreOpener-1c996d2a582a7ec43382d5a39bb54c78-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78/cf 2023-02-13 21:58:05,641 INFO [StoreOpener-1c996d2a582a7ec43382d5a39bb54c78-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1c996d2a582a7ec43382d5a39bb54c78 columnFamilyName cf 2023-02-13 21:58:05,642 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. 2023-02-13 21:58:05,642 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. 2023-02-13 21:58:05,642 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. 2023-02-13 21:58:05,643 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d1acdd49740dd716d5bec22786474a40, NAME => 'testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 21:58:05,643 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676325482126 d1acdd49740dd716d5bec22786474a40 2023-02-13 21:58:05,643 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:58:05,643 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for d1acdd49740dd716d5bec22786474a40 2023-02-13 21:58:05,643 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for d1acdd49740dd716d5bec22786474a40 2023-02-13 21:58:05,643 INFO [StoreOpener-1c996d2a582a7ec43382d5a39bb54c78-1] regionserver.HStore(310): Store=1c996d2a582a7ec43382d5a39bb54c78/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:58:05,645 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=f244b6e82e7a1752e9a3cb5f28dea2e9, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:58:05,645 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325485645"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325485645"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325485645"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325485645"}]},"ts":"1676325485645"} 2023-02-13 21:58:05,645 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for ef75de132134b515dbe11abe569ac292 2023-02-13 21:58:05,646 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=237, resume processing ppid=235 2023-02-13 21:58:05,646 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=237, ppid=235, state=SUCCESS; OpenRegionProcedure 00d462c2fbf63cefa67cf8e8f681246f, server=jenkins-hbase20.apache.org,39523,1676325451480 in 194 msec 2023-02-13 21:58:05,646 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78 2023-02-13 21:58:05,648 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=235, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=00d462c2fbf63cefa67cf8e8f681246f, ASSIGN in 365 msec 2023-02-13 21:58:05,649 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened ef75de132134b515dbe11abe569ac292; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65360447, jitterRate=-0.026053443551063538}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:58:05,649 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for ef75de132134b515dbe11abe569ac292: Running coprocessor pre-open hook at 1676325485621Writing region info on filesystem at 1676325485621Initializing all the Stores at 1676325485622 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325485622Cleaning up temporary data from old regions at 1676325485637 (+15 ms)Cleaning up detritus from prior splits at 1676325485639 (+2 ms)Running coprocessor post-open hooks at 1676325485649 (+10 ms)Region opened successfully at 1676325485649 2023-02-13 21:58:05,650 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=240, resume processing ppid=231 2023-02-13 21:58:05,650 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=240, ppid=231, state=SUCCESS; OpenRegionProcedure f244b6e82e7a1752e9a3cb5f28dea2e9, server=jenkins-hbase20.apache.org,45015,1676325451448 in 189 msec 2023-02-13 21:58:05,650 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292., pid=241, masterSystemTime=1676325485617 2023-02-13 21:58:05,650 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78 2023-02-13 21:58:05,652 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=231, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=f244b6e82e7a1752e9a3cb5f28dea2e9, ASSIGN in 369 msec 2023-02-13 21:58:05,652 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. 2023-02-13 21:58:05,652 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. 2023-02-13 21:58:05,652 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=ef75de132134b515dbe11abe569ac292, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:58:05,652 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. 2023-02-13 21:58:05,653 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325485652"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325485652"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325485652"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325485652"}]},"ts":"1676325485652"} 2023-02-13 21:58:05,653 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ac1d4ae96948f4de74aa3114b7308487, NAME => 'testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 21:58:05,653 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676325482126 ac1d4ae96948f4de74aa3114b7308487 2023-02-13 21:58:05,653 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:58:05,653 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for ac1d4ae96948f4de74aa3114b7308487 2023-02-13 21:58:05,653 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for ac1d4ae96948f4de74aa3114b7308487 2023-02-13 21:58:05,654 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 1c996d2a582a7ec43382d5a39bb54c78 2023-02-13 21:58:05,656 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 1c996d2a582a7ec43382d5a39bb54c78; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62955855, jitterRate=-0.06188465654850006}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:58:05,656 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 1c996d2a582a7ec43382d5a39bb54c78: Running coprocessor pre-open hook at 1676325485636Writing region info on filesystem at 1676325485636Initializing all the Stores at 1676325485637 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325485637Cleaning up temporary data from old regions at 1676325485651 (+14 ms)Cleaning up detritus from prior splits at 1676325485651Running coprocessor post-open hooks at 1676325485656 (+5 ms)Region opened successfully at 1676325485656 2023-02-13 21:58:05,656 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=241, resume processing ppid=234 2023-02-13 21:58:05,656 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=241, ppid=234, state=SUCCESS; OpenRegionProcedure ef75de132134b515dbe11abe569ac292, server=jenkins-hbase20.apache.org,36339,1676325451407 in 186 msec 2023-02-13 21:58:05,657 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78., pid=238, masterSystemTime=1676325485601 2023-02-13 21:58:05,658 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=234, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ef75de132134b515dbe11abe569ac292, ASSIGN in 375 msec 2023-02-13 21:58:05,659 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. 2023-02-13 21:58:05,659 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. 2023-02-13 21:58:05,660 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=1c996d2a582a7ec43382d5a39bb54c78, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:58:05,660 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676325485660"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325485660"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325485660"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325485660"}]},"ts":"1676325485660"} 2023-02-13 21:58:05,671 INFO [StoreOpener-d1acdd49740dd716d5bec22786474a40-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d1acdd49740dd716d5bec22786474a40 2023-02-13 21:58:05,671 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=238, resume processing ppid=230 2023-02-13 21:58:05,671 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=238, ppid=230, state=SUCCESS; OpenRegionProcedure 1c996d2a582a7ec43382d5a39bb54c78, server=jenkins-hbase20.apache.org,39523,1676325451480 in 214 msec 2023-02-13 21:58:05,674 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=230, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=1c996d2a582a7ec43382d5a39bb54c78, ASSIGN in 391 msec 2023-02-13 21:58:05,674 DEBUG [StoreOpener-d1acdd49740dd716d5bec22786474a40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40/cf 2023-02-13 21:58:05,674 DEBUG [StoreOpener-d1acdd49740dd716d5bec22786474a40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40/cf 2023-02-13 21:58:05,674 INFO [StoreOpener-d1acdd49740dd716d5bec22786474a40-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d1acdd49740dd716d5bec22786474a40 columnFamilyName cf 2023-02-13 21:58:05,675 INFO [StoreOpener-d1acdd49740dd716d5bec22786474a40-1] regionserver.HStore(310): Store=d1acdd49740dd716d5bec22786474a40/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:58:05,676 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40 2023-02-13 21:58:05,677 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40 2023-02-13 21:58:05,679 INFO [StoreOpener-ac1d4ae96948f4de74aa3114b7308487-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ac1d4ae96948f4de74aa3114b7308487 2023-02-13 21:58:05,680 DEBUG [StoreOpener-ac1d4ae96948f4de74aa3114b7308487-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487/cf 2023-02-13 21:58:05,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for d1acdd49740dd716d5bec22786474a40 2023-02-13 21:58:05,680 DEBUG [StoreOpener-ac1d4ae96948f4de74aa3114b7308487-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487/cf 2023-02-13 21:58:05,680 INFO [StoreOpener-ac1d4ae96948f4de74aa3114b7308487-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ac1d4ae96948f4de74aa3114b7308487 columnFamilyName cf 2023-02-13 21:58:05,680 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened d1acdd49740dd716d5bec22786474a40; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71040212, jitterRate=0.058581650257110596}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:58:05,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for d1acdd49740dd716d5bec22786474a40: Running coprocessor pre-open hook at 1676325485643Writing region info on filesystem at 1676325485643Initializing all the Stores at 1676325485653 (+10 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325485653Cleaning up temporary data from old regions at 1676325485677 (+24 ms)Cleaning up detritus from prior splits at 1676325485678 (+1 ms)Running coprocessor post-open hooks at 1676325485680 (+2 ms)Region opened successfully at 1676325485680 2023-02-13 21:58:05,681 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40., pid=236, masterSystemTime=1676325485596 2023-02-13 21:58:05,681 INFO [StoreOpener-ac1d4ae96948f4de74aa3114b7308487-1] regionserver.HStore(310): Store=ac1d4ae96948f4de74aa3114b7308487/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:58:05,682 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487 2023-02-13 21:58:05,683 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. 2023-02-13 21:58:05,683 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. 2023-02-13 21:58:05,683 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=d1acdd49740dd716d5bec22786474a40, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:58:05,683 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325485683"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325485683"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325485683"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325485683"}]},"ts":"1676325485683"} 2023-02-13 21:58:05,684 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487 2023-02-13 21:58:05,686 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=236, resume processing ppid=233 2023-02-13 21:58:05,686 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=236, ppid=233, state=SUCCESS; OpenRegionProcedure d1acdd49740dd716d5bec22786474a40, server=jenkins-hbase20.apache.org,45015,1676325451448 in 246 msec 2023-02-13 21:58:05,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for ac1d4ae96948f4de74aa3114b7308487 2023-02-13 21:58:05,688 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened ac1d4ae96948f4de74aa3114b7308487; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65878022, jitterRate=-0.018340975046157837}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:58:05,688 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=233, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=d1acdd49740dd716d5bec22786474a40, ASSIGN in 405 msec 2023-02-13 21:58:05,688 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for ac1d4ae96948f4de74aa3114b7308487: Running coprocessor pre-open hook at 1676325485653Writing region info on filesystem at 1676325485653Initializing all the Stores at 1676325485654 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325485654Cleaning up temporary data from old regions at 1676325485684 (+30 ms)Cleaning up detritus from prior splits at 1676325485685 (+1 ms)Running coprocessor post-open hooks at 1676325485688 (+3 ms)Region opened successfully at 1676325485688 2023-02-13 21:58:05,688 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487., pid=239, masterSystemTime=1676325485617 2023-02-13 21:58:05,689 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. 2023-02-13 21:58:05,689 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. 2023-02-13 21:58:05,690 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=ac1d4ae96948f4de74aa3114b7308487, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:58:05,691 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325485690"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325485690"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325485690"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325485690"}]},"ts":"1676325485690"} 2023-02-13 21:58:05,696 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=239, resume processing ppid=232 2023-02-13 21:58:05,696 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=239, ppid=232, state=SUCCESS; OpenRegionProcedure ac1d4ae96948f4de74aa3114b7308487, server=jenkins-hbase20.apache.org,36339,1676325451407 in 241 msec 2023-02-13 21:58:05,699 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=232, resume processing ppid=229 2023-02-13 21:58:05,699 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=232, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ac1d4ae96948f4de74aa3114b7308487, ASSIGN in 415 msec 2023-02-13 21:58:05,701 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676325482126","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325485701"}]},"ts":"1676325485701"} 2023-02-13 21:58:05,703 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676325482126, state=ENABLED in hbase:meta 2023-02-13 21:58:05,704 INFO [PEWorker-4] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1676325482126' was successfully enabled. 2023-02-13 21:58:05,707 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=229, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1676325482126 in 433 msec 2023-02-13 21:58:06,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-13 21:58:06,028 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1676325482126, procId: 229 completed 2023-02-13 21:58:06,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 21:58:06,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 21:58:06,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 21:58:06,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 21:58:06,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 21:58:06,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 21:58:06,045 DEBUG [Listener at localhost.localdomain/44075] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676325482126 2023-02-13 21:58:06,046 DEBUG [Listener at localhost.localdomain/44075] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. 2023-02-13 21:58:06,046 INFO [Listener at localhost.localdomain/44075] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 21:58:06,060 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 21:58:06,061 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 21:58:06,062 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 21:58:06,063 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 21:58:06,065 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 21:58:06,069 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676325482126 2023-02-13 21:58:06,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.HMaster$11(2405): Client=jenkins//148.251.75.209 disable testCorruptedSnapshot-1676325482126 2023-02-13 21:58:06,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=242, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676325482126 2023-02-13 21:58:06,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-13 21:58:06,075 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676325482126","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325486075"}]},"ts":"1676325486075"} 2023-02-13 21:58:06,077 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676325482126, state=DISABLING in hbase:meta 2023-02-13 21:58:06,078 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676325482126 to state=DISABLING 2023-02-13 21:58:06,080 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=1c996d2a582a7ec43382d5a39bb54c78, UNASSIGN}, {pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=f244b6e82e7a1752e9a3cb5f28dea2e9, UNASSIGN}, {pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ac1d4ae96948f4de74aa3114b7308487, UNASSIGN}, {pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=d1acdd49740dd716d5bec22786474a40, UNASSIGN}, {pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ef75de132134b515dbe11abe569ac292, UNASSIGN}, {pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=00d462c2fbf63cefa67cf8e8f681246f, UNASSIGN}] 2023-02-13 21:58:06,081 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=00d462c2fbf63cefa67cf8e8f681246f, UNASSIGN 2023-02-13 21:58:06,081 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ef75de132134b515dbe11abe569ac292, UNASSIGN 2023-02-13 21:58:06,081 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=d1acdd49740dd716d5bec22786474a40, UNASSIGN 2023-02-13 21:58:06,081 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ac1d4ae96948f4de74aa3114b7308487, UNASSIGN 2023-02-13 21:58:06,082 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=f244b6e82e7a1752e9a3cb5f28dea2e9, UNASSIGN 2023-02-13 21:58:06,083 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=00d462c2fbf63cefa67cf8e8f681246f, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:58:06,083 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676325486083"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325486083"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325486083"}]},"ts":"1676325486083"} 2023-02-13 21:58:06,083 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=ef75de132134b515dbe11abe569ac292, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:58:06,083 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325486083"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325486083"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325486083"}]},"ts":"1676325486083"} 2023-02-13 21:58:06,084 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=d1acdd49740dd716d5bec22786474a40, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:58:06,084 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325486084"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325486084"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325486084"}]},"ts":"1676325486084"} 2023-02-13 21:58:06,084 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=f244b6e82e7a1752e9a3cb5f28dea2e9, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:58:06,084 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325486084"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325486084"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325486084"}]},"ts":"1676325486084"} 2023-02-13 21:58:06,084 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=ac1d4ae96948f4de74aa3114b7308487, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:58:06,084 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325486084"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325486084"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325486084"}]},"ts":"1676325486084"} 2023-02-13 21:58:06,092 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=249, ppid=248, state=RUNNABLE; CloseRegionProcedure 00d462c2fbf63cefa67cf8e8f681246f, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:58:06,094 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=250, ppid=247, state=RUNNABLE; CloseRegionProcedure ef75de132134b515dbe11abe569ac292, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:58:06,095 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=251, ppid=246, state=RUNNABLE; CloseRegionProcedure d1acdd49740dd716d5bec22786474a40, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:58:06,096 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=252, ppid=244, state=RUNNABLE; CloseRegionProcedure f244b6e82e7a1752e9a3cb5f28dea2e9, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:58:06,097 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=253, ppid=245, state=RUNNABLE; CloseRegionProcedure ac1d4ae96948f4de74aa3114b7308487, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:58:06,098 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=1c996d2a582a7ec43382d5a39bb54c78, UNASSIGN 2023-02-13 21:58:06,099 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=1c996d2a582a7ec43382d5a39bb54c78, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:58:06,099 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676325486099"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325486099"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325486099"}]},"ts":"1676325486099"} 2023-02-13 21:58:06,101 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=254, ppid=243, state=RUNNABLE; CloseRegionProcedure 1c996d2a582a7ec43382d5a39bb54c78, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:58:06,246 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 1c996d2a582a7ec43382d5a39bb54c78 2023-02-13 21:58:06,246 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 1c996d2a582a7ec43382d5a39bb54c78, disabling compactions & flushes 2023-02-13 21:58:06,246 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. 2023-02-13 21:58:06,246 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. 2023-02-13 21:58:06,246 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. after waiting 0 ms 2023-02-13 21:58:06,246 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. 2023-02-13 21:58:06,246 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing 1c996d2a582a7ec43382d5a39bb54c78 1/1 column families, dataSize=1.83 KB heapSize=4.19 KB 2023-02-13 21:58:06,249 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close ef75de132134b515dbe11abe569ac292 2023-02-13 21:58:06,249 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close f244b6e82e7a1752e9a3cb5f28dea2e9 2023-02-13 21:58:06,249 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing ef75de132134b515dbe11abe569ac292, disabling compactions & flushes 2023-02-13 21:58:06,249 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. 2023-02-13 21:58:06,249 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing f244b6e82e7a1752e9a3cb5f28dea2e9, disabling compactions & flushes 2023-02-13 21:58:06,249 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. 2023-02-13 21:58:06,249 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. 2023-02-13 21:58:06,249 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. after waiting 0 ms 2023-02-13 21:58:06,249 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. 2023-02-13 21:58:06,249 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. 2023-02-13 21:58:06,250 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. after waiting 0 ms 2023-02-13 21:58:06,250 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. 2023-02-13 21:58:06,250 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing ef75de132134b515dbe11abe569ac292 1/1 column families, dataSize=2.03 KB heapSize=4.61 KB 2023-02-13 21:58:06,250 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing f244b6e82e7a1752e9a3cb5f28dea2e9 1/1 column families, dataSize=2.16 KB heapSize=4.89 KB 2023-02-13 21:58:06,274 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202302136ca58bd8a658451cb2d779c5510856a9 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/d41d8cd98f00b204e9800998ecf8427e202302136ca58bd8a658451cb2d779c5510856a9 2023-02-13 21:58:06,276 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b2023021391c649b08d6f4b43b12afda4c89899e7 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/c4ca4238a0b923820dcc509a6f75849b2023021391c649b08d6f4b43b12afda4c89899e7 2023-02-13 21:58:06,278 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c20230213ba864453080947f097171ffb4373beab to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/a87ff679a2f3e71d9181a67b7542122c20230213ba864453080947f097171ffb4373beab 2023-02-13 21:58:06,295 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78/.tmp/cf/f68183f886e84213becb66fd697bf854 2023-02-13 21:58:06,295 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9/.tmp/cf/b25fb604e9074f249714afdbfac4c6d3 2023-02-13 21:58:06,297 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292/.tmp/cf/8e37f961530e4488a37efc85ab1c86ac 2023-02-13 21:58:06,298 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78/.tmp/cf/f68183f886e84213becb66fd697bf854 as hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78/cf/f68183f886e84213becb66fd697bf854 2023-02-13 21:58:06,299 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9/.tmp/cf/b25fb604e9074f249714afdbfac4c6d3 as hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9/cf/b25fb604e9074f249714afdbfac4c6d3 2023-02-13 21:58:06,301 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292/.tmp/cf/8e37f961530e4488a37efc85ab1c86ac as hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292/cf/8e37f961530e4488a37efc85ab1c86ac 2023-02-13 21:58:06,301 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78/cf/f68183f886e84213becb66fd697bf854, entries=28, sequenceid=8, filesize=9.7 K 2023-02-13 21:58:06,301 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9/cf/b25fb604e9074f249714afdbfac4c6d3, entries=33, sequenceid=8, filesize=10.6 K 2023-02-13 21:58:06,302 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.83 KB/1874, heapSize ~4.17 KB/4272, currentSize=0 B/0 for 1c996d2a582a7ec43382d5a39bb54c78 in 56ms, sequenceid=8, compaction requested=false 2023-02-13 21:58:06,302 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testCorruptedSnapshot-1676325482126' 2023-02-13 21:58:06,302 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for f244b6e82e7a1752e9a3cb5f28dea2e9 in 52ms, sequenceid=8, compaction requested=false 2023-02-13 21:58:06,302 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testCorruptedSnapshot-1676325482126' 2023-02-13 21:58:06,306 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292/cf/8e37f961530e4488a37efc85ab1c86ac, entries=31, sequenceid=8, filesize=10.2 K 2023-02-13 21:58:06,307 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for ef75de132134b515dbe11abe569ac292 in 57ms, sequenceid=8, compaction requested=false 2023-02-13 21:58:06,307 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 21:58:06,307 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 21:58:06,307 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. 2023-02-13 21:58:06,307 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. 2023-02-13 21:58:06,308 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for f244b6e82e7a1752e9a3cb5f28dea2e9: Waiting for close lock at 1676325486249Running coprocessor pre-close hooks at 1676325486249Disabling compacts and flushes for region at 1676325486249Disabling writes for close at 1676325486250 (+1 ms)Obtaining lock to block concurrent updates at 1676325486250Preparing flush snapshotting stores in f244b6e82e7a1752e9a3cb5f28dea2e9 at 1676325486250Finished memstore snapshotting testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9., syncing WAL and waiting on mvcc, flushsize=dataSize=2209, getHeapSize=4992, getOffHeapSize=0, getCellsCount=33 at 1676325486250Flushing stores of testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. at 1676325486251 (+1 ms)Flushing f244b6e82e7a1752e9a3cb5f28dea2e9/cf: creating writer at 1676325486252 (+1 ms)Flushing mob file f244b6e82e7a1752e9a3cb5f28dea2e9/cf: appending metadata at 1676325486263 (+11 ms)Flushing mob file f244b6e82e7a1752e9a3cb5f28dea2e9/cf: closing flushed file at 1676325486263Flushing f244b6e82e7a1752e9a3cb5f28dea2e9/cf: appending metadata at 1676325486277 (+14 ms)Flushing f244b6e82e7a1752e9a3cb5f28dea2e9/cf: closing flushed file at 1676325486277Flushing f244b6e82e7a1752e9a3cb5f28dea2e9/cf: reopening flushed file at 1676325486299 (+22 ms)Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for f244b6e82e7a1752e9a3cb5f28dea2e9 in 52ms, sequenceid=8, compaction requested=false at 1676325486302 (+3 ms)Writing region close event to WAL at 1676325486304 (+2 ms)Running coprocessor post-close hooks at 1676325486307 (+3 ms)Closed at 1676325486307 2023-02-13 21:58:06,308 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 1c996d2a582a7ec43382d5a39bb54c78: Waiting for close lock at 1676325486246Running coprocessor pre-close hooks at 1676325486246Disabling compacts and flushes for region at 1676325486246Disabling writes for close at 1676325486246Obtaining lock to block concurrent updates at 1676325486247 (+1 ms)Preparing flush snapshotting stores in 1c996d2a582a7ec43382d5a39bb54c78 at 1676325486247Finished memstore snapshotting testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78., syncing WAL and waiting on mvcc, flushsize=dataSize=1874, getHeapSize=4272, getOffHeapSize=0, getCellsCount=28 at 1676325486247Flushing stores of testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. at 1676325486247Flushing 1c996d2a582a7ec43382d5a39bb54c78/cf: creating writer at 1676325486248 (+1 ms)Flushing mob file 1c996d2a582a7ec43382d5a39bb54c78/cf: appending metadata at 1676325486254 (+6 ms)Flushing mob file 1c996d2a582a7ec43382d5a39bb54c78/cf: closing flushed file at 1676325486254Flushing 1c996d2a582a7ec43382d5a39bb54c78/cf: appending metadata at 1676325486275 (+21 ms)Flushing 1c996d2a582a7ec43382d5a39bb54c78/cf: closing flushed file at 1676325486275Flushing 1c996d2a582a7ec43382d5a39bb54c78/cf: reopening flushed file at 1676325486299 (+24 ms)Finished flush of dataSize ~1.83 KB/1874, heapSize ~4.17 KB/4272, currentSize=0 B/0 for 1c996d2a582a7ec43382d5a39bb54c78 in 56ms, sequenceid=8, compaction requested=false at 1676325486302 (+3 ms)Writing region close event to WAL at 1676325486304 (+2 ms)Running coprocessor post-close hooks at 1676325486307 (+3 ms)Closed at 1676325486307 2023-02-13 21:58:06,309 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 1c996d2a582a7ec43382d5a39bb54c78 2023-02-13 21:58:06,309 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 00d462c2fbf63cefa67cf8e8f681246f 2023-02-13 21:58:06,309 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 00d462c2fbf63cefa67cf8e8f681246f, disabling compactions & flushes 2023-02-13 21:58:06,309 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. 2023-02-13 21:58:06,309 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. 2023-02-13 21:58:06,309 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. after waiting 0 ms 2023-02-13 21:58:06,309 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. 2023-02-13 21:58:06,309 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing 00d462c2fbf63cefa67cf8e8f681246f 1/1 column families, dataSize=22.57 KB heapSize=48.77 KB 2023-02-13 21:58:06,309 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=1c996d2a582a7ec43382d5a39bb54c78, regionState=CLOSED 2023-02-13 21:58:06,309 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676325486309"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325486309"}]},"ts":"1676325486309"} 2023-02-13 21:58:06,309 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed f244b6e82e7a1752e9a3cb5f28dea2e9 2023-02-13 21:58:06,310 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close d1acdd49740dd716d5bec22786474a40 2023-02-13 21:58:06,310 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing d1acdd49740dd716d5bec22786474a40, disabling compactions & flushes 2023-02-13 21:58:06,310 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. 2023-02-13 21:58:06,310 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. 2023-02-13 21:58:06,310 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. after waiting 0 ms 2023-02-13 21:58:06,310 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. 2023-02-13 21:58:06,311 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing d1acdd49740dd716d5bec22786474a40 1/1 column families, dataSize=1.96 KB heapSize=4.47 KB 2023-02-13 21:58:06,311 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=f244b6e82e7a1752e9a3cb5f28dea2e9, regionState=CLOSED 2023-02-13 21:58:06,311 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325486311"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325486311"}]},"ts":"1676325486311"} 2023-02-13 21:58:06,311 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 21:58:06,312 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. 2023-02-13 21:58:06,313 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for ef75de132134b515dbe11abe569ac292: Waiting for close lock at 1676325486249Running coprocessor pre-close hooks at 1676325486249Disabling compacts and flushes for region at 1676325486249Disabling writes for close at 1676325486249Obtaining lock to block concurrent updates at 1676325486250 (+1 ms)Preparing flush snapshotting stores in ef75de132134b515dbe11abe569ac292 at 1676325486250Finished memstore snapshotting testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292., syncing WAL and waiting on mvcc, flushsize=dataSize=2075, getHeapSize=4704, getOffHeapSize=0, getCellsCount=31 at 1676325486250Flushing stores of testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. at 1676325486251 (+1 ms)Flushing ef75de132134b515dbe11abe569ac292/cf: creating writer at 1676325486252 (+1 ms)Flushing mob file ef75de132134b515dbe11abe569ac292/cf: appending metadata at 1676325486259 (+7 ms)Flushing mob file ef75de132134b515dbe11abe569ac292/cf: closing flushed file at 1676325486259Flushing ef75de132134b515dbe11abe569ac292/cf: appending metadata at 1676325486278 (+19 ms)Flushing ef75de132134b515dbe11abe569ac292/cf: closing flushed file at 1676325486278Flushing ef75de132134b515dbe11abe569ac292/cf: reopening flushed file at 1676325486301 (+23 ms)Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for ef75de132134b515dbe11abe569ac292 in 57ms, sequenceid=8, compaction requested=false at 1676325486307 (+6 ms)Writing region close event to WAL at 1676325486307Running coprocessor post-close hooks at 1676325486312 (+5 ms)Closed at 1676325486312 2023-02-13 21:58:06,315 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed ef75de132134b515dbe11abe569ac292 2023-02-13 21:58:06,315 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close ac1d4ae96948f4de74aa3114b7308487 2023-02-13 21:58:06,315 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=254, resume processing ppid=243 2023-02-13 21:58:06,315 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing ac1d4ae96948f4de74aa3114b7308487, disabling compactions & flushes 2023-02-13 21:58:06,315 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=254, ppid=243, state=SUCCESS; CloseRegionProcedure 1c996d2a582a7ec43382d5a39bb54c78, server=jenkins-hbase20.apache.org,39523,1676325451480 in 210 msec 2023-02-13 21:58:06,315 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. 2023-02-13 21:58:06,315 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. 2023-02-13 21:58:06,315 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. after waiting 0 ms 2023-02-13 21:58:06,315 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. 2023-02-13 21:58:06,315 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing ac1d4ae96948f4de74aa3114b7308487 1/1 column families, dataSize=2.16 KB heapSize=4.89 KB 2023-02-13 21:58:06,315 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=252, resume processing ppid=244 2023-02-13 21:58:06,315 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=ef75de132134b515dbe11abe569ac292, regionState=CLOSED 2023-02-13 21:58:06,316 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325486315"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325486315"}]},"ts":"1676325486315"} 2023-02-13 21:58:06,315 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=252, ppid=244, state=SUCCESS; CloseRegionProcedure f244b6e82e7a1752e9a3cb5f28dea2e9, server=jenkins-hbase20.apache.org,45015,1676325451448 in 218 msec 2023-02-13 21:58:06,316 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=243, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=1c996d2a582a7ec43382d5a39bb54c78, UNASSIGN in 236 msec 2023-02-13 21:58:06,317 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=244, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=f244b6e82e7a1752e9a3cb5f28dea2e9, UNASSIGN in 236 msec 2023-02-13 21:58:06,318 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=250, resume processing ppid=247 2023-02-13 21:58:06,318 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=250, ppid=247, state=SUCCESS; CloseRegionProcedure ef75de132134b515dbe11abe569ac292, server=jenkins-hbase20.apache.org,36339,1676325451407 in 223 msec 2023-02-13 21:58:06,320 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=247, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ef75de132134b515dbe11abe569ac292, UNASSIGN in 239 msec 2023-02-13 21:58:06,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-13 21:58:06,330 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230213683c128b9eaf468a8ade77032e7ee80a to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213683c128b9eaf468a8ade77032e7ee80a 2023-02-13 21:58:06,346 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d520230213afc9688d0a4e4dbfbacc2eb854736faa to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/e4da3b7fbbce2345d7772b0674a318d520230213afc9688d0a4e4dbfbacc2eb854736faa 2023-02-13 21:58:06,346 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c202302135f82ed0773c04f148549aaf42ad53f50 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/c81e728d9d4c2f636f067f89cc14862c202302135f82ed0773c04f148549aaf42ad53f50 2023-02-13 21:58:06,348 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40/.tmp/cf/278f4c4405a04414a1eb5319d07c2091 2023-02-13 21:58:06,352 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40/.tmp/cf/278f4c4405a04414a1eb5319d07c2091 as hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40/cf/278f4c4405a04414a1eb5319d07c2091 2023-02-13 21:58:06,354 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40/cf/278f4c4405a04414a1eb5319d07c2091, entries=30, sequenceid=8, filesize=10.1 K 2023-02-13 21:58:06,355 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for d1acdd49740dd716d5bec22786474a40 in 44ms, sequenceid=8, compaction requested=false 2023-02-13 21:58:06,356 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487/.tmp/cf/a015f28c44be4a2f913a58fe70ca099e 2023-02-13 21:58:06,356 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=22.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/.tmp/cf/ead9f71bf7144e80bc30d97efc70516f 2023-02-13 21:58:06,358 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 21:58:06,358 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. 2023-02-13 21:58:06,359 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for d1acdd49740dd716d5bec22786474a40: Waiting for close lock at 1676325486310Running coprocessor pre-close hooks at 1676325486310Disabling compacts and flushes for region at 1676325486310Disabling writes for close at 1676325486310Obtaining lock to block concurrent updates at 1676325486311 (+1 ms)Preparing flush snapshotting stores in d1acdd49740dd716d5bec22786474a40 at 1676325486311Finished memstore snapshotting testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40., syncing WAL and waiting on mvcc, flushsize=dataSize=2008, getHeapSize=4560, getOffHeapSize=0, getCellsCount=30 at 1676325486311Flushing stores of testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. at 1676325486311Flushing d1acdd49740dd716d5bec22786474a40/cf: creating writer at 1676325486313 (+2 ms)Flushing mob file d1acdd49740dd716d5bec22786474a40/cf: appending metadata at 1676325486318 (+5 ms)Flushing mob file d1acdd49740dd716d5bec22786474a40/cf: closing flushed file at 1676325486318Flushing d1acdd49740dd716d5bec22786474a40/cf: appending metadata at 1676325486331 (+13 ms)Flushing d1acdd49740dd716d5bec22786474a40/cf: closing flushed file at 1676325486331Flushing d1acdd49740dd716d5bec22786474a40/cf: reopening flushed file at 1676325486352 (+21 ms)Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for d1acdd49740dd716d5bec22786474a40 in 44ms, sequenceid=8, compaction requested=false at 1676325486355 (+3 ms)Writing region close event to WAL at 1676325486356 (+1 ms)Running coprocessor post-close hooks at 1676325486358 (+2 ms)Closed at 1676325486358 2023-02-13 21:58:06,359 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/.tmp/cf/ead9f71bf7144e80bc30d97efc70516f as hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/cf/ead9f71bf7144e80bc30d97efc70516f 2023-02-13 21:58:06,359 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487/.tmp/cf/a015f28c44be4a2f913a58fe70ca099e as hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487/cf/a015f28c44be4a2f913a58fe70ca099e 2023-02-13 21:58:06,360 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed d1acdd49740dd716d5bec22786474a40 2023-02-13 21:58:06,360 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=d1acdd49740dd716d5bec22786474a40, regionState=CLOSED 2023-02-13 21:58:06,360 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325486360"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325486360"}]},"ts":"1676325486360"} 2023-02-13 21:58:06,362 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487/cf/a015f28c44be4a2f913a58fe70ca099e, entries=33, sequenceid=8, filesize=10.6 K 2023-02-13 21:58:06,362 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/cf/ead9f71bf7144e80bc30d97efc70516f, entries=345, sequenceid=8, filesize=64.4 K 2023-02-13 21:58:06,362 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=251, resume processing ppid=246 2023-02-13 21:58:06,363 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=251, ppid=246, state=SUCCESS; CloseRegionProcedure d1acdd49740dd716d5bec22786474a40, server=jenkins-hbase20.apache.org,45015,1676325451448 in 266 msec 2023-02-13 21:58:06,363 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for ac1d4ae96948f4de74aa3114b7308487 in 48ms, sequenceid=8, compaction requested=false 2023-02-13 21:58:06,363 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~22.57 KB/23111, heapSize ~48.75 KB/49920, currentSize=0 B/0 for 00d462c2fbf63cefa67cf8e8f681246f in 54ms, sequenceid=8, compaction requested=false 2023-02-13 21:58:06,364 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=246, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=d1acdd49740dd716d5bec22786474a40, UNASSIGN in 284 msec 2023-02-13 21:58:06,365 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 21:58:06,365 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 21:58:06,366 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. 2023-02-13 21:58:06,366 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. 2023-02-13 21:58:06,366 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for ac1d4ae96948f4de74aa3114b7308487: Waiting for close lock at 1676325486315Running coprocessor pre-close hooks at 1676325486315Disabling compacts and flushes for region at 1676325486315Disabling writes for close at 1676325486315Obtaining lock to block concurrent updates at 1676325486315Preparing flush snapshotting stores in ac1d4ae96948f4de74aa3114b7308487 at 1676325486315Finished memstore snapshotting testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487., syncing WAL and waiting on mvcc, flushsize=dataSize=2209, getHeapSize=4992, getOffHeapSize=0, getCellsCount=33 at 1676325486315Flushing stores of testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. at 1676325486316 (+1 ms)Flushing ac1d4ae96948f4de74aa3114b7308487/cf: creating writer at 1676325486316Flushing mob file ac1d4ae96948f4de74aa3114b7308487/cf: appending metadata at 1676325486320 (+4 ms)Flushing mob file ac1d4ae96948f4de74aa3114b7308487/cf: closing flushed file at 1676325486320Flushing ac1d4ae96948f4de74aa3114b7308487/cf: appending metadata at 1676325486347 (+27 ms)Flushing ac1d4ae96948f4de74aa3114b7308487/cf: closing flushed file at 1676325486347Flushing ac1d4ae96948f4de74aa3114b7308487/cf: reopening flushed file at 1676325486360 (+13 ms)Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for ac1d4ae96948f4de74aa3114b7308487 in 48ms, sequenceid=8, compaction requested=false at 1676325486363 (+3 ms)Writing region close event to WAL at 1676325486363Running coprocessor post-close hooks at 1676325486365 (+2 ms)Closed at 1676325486366 (+1 ms) 2023-02-13 21:58:06,366 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 00d462c2fbf63cefa67cf8e8f681246f: Waiting for close lock at 1676325486309Running coprocessor pre-close hooks at 1676325486309Disabling compacts and flushes for region at 1676325486309Disabling writes for close at 1676325486309Obtaining lock to block concurrent updates at 1676325486309Preparing flush snapshotting stores in 00d462c2fbf63cefa67cf8e8f681246f at 1676325486309Finished memstore snapshotting testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f., syncing WAL and waiting on mvcc, flushsize=dataSize=23111, getHeapSize=49920, getOffHeapSize=0, getCellsCount=345 at 1676325486309Flushing stores of testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. at 1676325486310 (+1 ms)Flushing 00d462c2fbf63cefa67cf8e8f681246f/cf: creating writer at 1676325486311 (+1 ms)Flushing mob file 00d462c2fbf63cefa67cf8e8f681246f/cf: appending metadata at 1676325486320 (+9 ms)Flushing mob file 00d462c2fbf63cefa67cf8e8f681246f/cf: closing flushed file at 1676325486320Flushing 00d462c2fbf63cefa67cf8e8f681246f/cf: appending metadata at 1676325486347 (+27 ms)Flushing 00d462c2fbf63cefa67cf8e8f681246f/cf: closing flushed file at 1676325486347Flushing 00d462c2fbf63cefa67cf8e8f681246f/cf: reopening flushed file at 1676325486359 (+12 ms)Finished flush of dataSize ~22.57 KB/23111, heapSize ~48.75 KB/49920, currentSize=0 B/0 for 00d462c2fbf63cefa67cf8e8f681246f in 54ms, sequenceid=8, compaction requested=false at 1676325486363 (+4 ms)Writing region close event to WAL at 1676325486363Running coprocessor post-close hooks at 1676325486366 (+3 ms)Closed at 1676325486366 2023-02-13 21:58:06,367 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 00d462c2fbf63cefa67cf8e8f681246f 2023-02-13 21:58:06,367 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=00d462c2fbf63cefa67cf8e8f681246f, regionState=CLOSED 2023-02-13 21:58:06,367 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676325486367"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325486367"}]},"ts":"1676325486367"} 2023-02-13 21:58:06,367 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed ac1d4ae96948f4de74aa3114b7308487 2023-02-13 21:58:06,367 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=ac1d4ae96948f4de74aa3114b7308487, regionState=CLOSED 2023-02-13 21:58:06,367 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325486367"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325486367"}]},"ts":"1676325486367"} 2023-02-13 21:58:06,369 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=249, resume processing ppid=248 2023-02-13 21:58:06,369 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=249, ppid=248, state=SUCCESS; CloseRegionProcedure 00d462c2fbf63cefa67cf8e8f681246f, server=jenkins-hbase20.apache.org,39523,1676325451480 in 277 msec 2023-02-13 21:58:06,369 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=253, resume processing ppid=245 2023-02-13 21:58:06,369 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=253, ppid=245, state=SUCCESS; CloseRegionProcedure ac1d4ae96948f4de74aa3114b7308487, server=jenkins-hbase20.apache.org,36339,1676325451407 in 271 msec 2023-02-13 21:58:06,369 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=248, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=00d462c2fbf63cefa67cf8e8f681246f, UNASSIGN in 290 msec 2023-02-13 21:58:06,371 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=245, resume processing ppid=242 2023-02-13 21:58:06,371 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=245, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ac1d4ae96948f4de74aa3114b7308487, UNASSIGN in 290 msec 2023-02-13 21:58:06,371 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676325482126","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325486371"}]},"ts":"1676325486371"} 2023-02-13 21:58:06,372 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676325482126, state=DISABLED in hbase:meta 2023-02-13 21:58:06,373 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676325482126 to state=DISABLED 2023-02-13 21:58:06,374 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=242, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676325482126 in 304 msec 2023-02-13 21:58:06,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-13 21:58:06,826 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676325482126, procId: 242 completed 2023-02-13 21:58:06,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1580): Client=jenkins//148.251.75.209 snapshot request for:{ ss=snaptb0-1676325482126 table=testCorruptedSnapshot-1676325482126 type=FLUSH ttl=0 } 2023-02-13 21:58:06,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676325486827 (current time:1676325486827). 2023-02-13 21:58:06,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 21:58:06,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] zookeeper.ReadOnlyZKClient(139): Connect 0x16e63d44 to 127.0.0.1:58580 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 21:58:06,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5d5d8011, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 21:58:06,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 21:58:06,840 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60268, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 21:58:06,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x16e63d44 to 127.0.0.1:58580 2023-02-13 21:58:06,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 21:58:06,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 21:58:06,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1676325482126 table=testCorruptedSnapshot-1676325482126 type=FLUSH ttl=0 } 2023-02-13 21:58:06,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676325482126, type=EXCLUSIVE 2023-02-13 21:58:06,844 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676325482126, type=EXCLUSIVE 2023-02-13 21:58:06,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1676325482126 table=testCorruptedSnapshot-1676325482126 type=FLUSH ttl=0 } 2023-02-13 21:58:06,845 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=255, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676325482126, type=EXCLUSIVE; timeout=600000, timestamp=1676326086845 2023-02-13 21:58:06,845 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1676325482126 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676325482126 2023-02-13 21:58:06,845 DEBUG [Listener at localhost.localdomain/44075] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1676325482126 table=testCorruptedSnapshot-1676325482126 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 21:58:06,845 DEBUG [Listener at localhost.localdomain/44075] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 21:58:06,852 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676325482126 table=testCorruptedSnapshot-1676325482126 type=DISABLED ttl=0 } 2023-02-13 21:58:06,858 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:58:06,858 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:58:06,858 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:58:06,858 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:58:06,858 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:58:06,858 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:58:06,858 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:58:06,858 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:58:06,859 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:58:06,859 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:58:06,859 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:58:06,859 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 21:58:06,859 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:58:06,859 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 21:58:06,860 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78/cf/f68183f886e84213becb66fd697bf854] hfiles 2023-02-13 21:58:06,860 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78/cf/f68183f886e84213becb66fd697bf854 2023-02-13 21:58:06,861 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487/cf/a015f28c44be4a2f913a58fe70ca099e] hfiles 2023-02-13 21:58:06,861 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487/cf/a015f28c44be4a2f913a58fe70ca099e 2023-02-13 21:58:06,861 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292/cf/8e37f961530e4488a37efc85ab1c86ac] hfiles 2023-02-13 21:58:06,861 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292/cf/8e37f961530e4488a37efc85ab1c86ac 2023-02-13 21:58:06,861 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40/cf/278f4c4405a04414a1eb5319d07c2091] hfiles 2023-02-13 21:58:06,861 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/cf/ead9f71bf7144e80bc30d97efc70516f] hfiles 2023-02-13 21:58:06,861 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40/cf/278f4c4405a04414a1eb5319d07c2091 2023-02-13 21:58:06,861 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/cf/ead9f71bf7144e80bc30d97efc70516f 2023-02-13 21:58:06,862 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9/cf/b25fb604e9074f249714afdbfac4c6d3] hfiles 2023-02-13 21:58:06,862 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9/cf/b25fb604e9074f249714afdbfac4c6d3 2023-02-13 21:58:06,863 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/a87ff679a2f3e71d9181a67b7542122c20230213ba864453080947f097171ffb4373beab, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/c4ca4238a0b923820dcc509a6f75849b2023021391c649b08d6f4b43b12afda4c89899e7, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/c81e728d9d4c2f636f067f89cc14862c202302135f82ed0773c04f148549aaf42ad53f50, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/d41d8cd98f00b204e9800998ecf8427e202302136ca58bd8a658451cb2d779c5510856a9, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/e4da3b7fbbce2345d7772b0674a318d520230213afc9688d0a4e4dbfbacc2eb854736faa, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213683c128b9eaf468a8ade77032e7ee80a] hfiles 2023-02-13 21:58:06,863 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/a87ff679a2f3e71d9181a67b7542122c20230213ba864453080947f097171ffb4373beab 2023-02-13 21:58:06,863 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/c4ca4238a0b923820dcc509a6f75849b2023021391c649b08d6f4b43b12afda4c89899e7 2023-02-13 21:58:06,863 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/c81e728d9d4c2f636f067f89cc14862c202302135f82ed0773c04f148549aaf42ad53f50 2023-02-13 21:58:06,863 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/d41d8cd98f00b204e9800998ecf8427e202302136ca58bd8a658451cb2d779c5510856a9 2023-02-13 21:58:06,863 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/e4da3b7fbbce2345d7772b0674a318d520230213afc9688d0a4e4dbfbacc2eb854736faa 2023-02-13 21:58:06,863 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213683c128b9eaf468a8ade77032e7ee80a 2023-02-13 21:58:06,882 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1676325482126 table=testCorruptedSnapshot-1676325482126 type=DISABLED ttl=0 } as finished. 2023-02-13 21:58:06,882 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1676325482126 2023-02-13 21:58:06,883 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/snaptb0-1676325482126 2023-02-13 21:58:06,897 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/snaptb0-1676325482126 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/snaptb0-1676325482126 2023-02-13 21:58:06,919 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1676325482126 of table testCorruptedSnapshot-1676325482126 completed 2023-02-13 21:58:06,919 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/snaptb0-1676325482126 2023-02-13 21:58:06,919 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.hbase-snapshot/.tmp/snaptb0-1676325482126 2023-02-13 21:58:06,919 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1676325482126 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676325482126 at 1676325486845Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676325482126 table=testCorruptedSnapshot-1676325482126 type=DISABLED ttl=0 } at 1676325486852 (+7 ms)Consolidate snapshot: snaptb0-1676325482126 at 1676325486882 (+30 ms)Loading Region manifests for snaptb0-1676325482126 at 1676325486883 (+1 ms)Writing data manifest for snaptb0-1676325482126 at 1676325486885 (+2 ms)Verifying snapshot: snaptb0-1676325482126 at 1676325486891 (+6 ms)Snapshot snaptb0-1676325482126 of table testCorruptedSnapshot-1676325482126 completed at 1676325486919 (+28 ms) 2023-02-13 21:58:06,920 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=255, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676325482126, type=EXCLUSIVE 2023-02-13 21:58:06,921 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=255, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676325482126, type=EXCLUSIVE in 77 msec 2023-02-13 21:58:07,095 DEBUG [Listener at localhost.localdomain/44075] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 21:58:07,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676325482126 table=testCorruptedSnapshot-1676325482126 type=FLUSH ttl=0 } is done 2023-02-13 21:58:07,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1676325482126 table=testCorruptedSnapshot-1676325482126 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 21:58:07,098 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1676325482126 2023-02-13 21:58:07,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.HMaster$10(2372): Client=jenkins//148.251.75.209 enable testCorruptedSnapshot-1676325482126 2023-02-13 21:58:07,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=256, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1676325482126 2023-02-13 21:58:07,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-13 21:58:07,104 INFO [PEWorker-5] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1676325482126 2023-02-13 21:58:07,104 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676325482126","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325487104"}]},"ts":"1676325487104"} 2023-02-13 21:58:07,106 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676325482126, state=ENABLING in hbase:meta 2023-02-13 21:58:07,109 DEBUG [PEWorker-5] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 21:58:07,109 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=1c996d2a582a7ec43382d5a39bb54c78, ASSIGN}, {pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=f244b6e82e7a1752e9a3cb5f28dea2e9, ASSIGN}, {pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ac1d4ae96948f4de74aa3114b7308487, ASSIGN}, {pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=d1acdd49740dd716d5bec22786474a40, ASSIGN}, {pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ef75de132134b515dbe11abe569ac292, ASSIGN}, {pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=00d462c2fbf63cefa67cf8e8f681246f, ASSIGN}] 2023-02-13 21:58:07,110 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ac1d4ae96948f4de74aa3114b7308487, ASSIGN 2023-02-13 21:58:07,110 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=f244b6e82e7a1752e9a3cb5f28dea2e9, ASSIGN 2023-02-13 21:58:07,110 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=1c996d2a582a7ec43382d5a39bb54c78, ASSIGN 2023-02-13 21:58:07,110 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=d1acdd49740dd716d5bec22786474a40, ASSIGN 2023-02-13 21:58:07,111 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:58:07,111 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ac1d4ae96948f4de74aa3114b7308487, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,36339,1676325451407; forceNewPlan=false, retain=true 2023-02-13 21:58:07,111 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:58:07,111 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:58:07,111 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:58:07,111 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=f244b6e82e7a1752e9a3cb5f28dea2e9, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,45015,1676325451448; forceNewPlan=false, retain=true 2023-02-13 21:58:07,111 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=d1acdd49740dd716d5bec22786474a40, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,45015,1676325451448; forceNewPlan=false, retain=true 2023-02-13 21:58:07,111 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=1c996d2a582a7ec43382d5a39bb54c78, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,39523,1676325451480; forceNewPlan=false, retain=true 2023-02-13 21:58:07,111 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=00d462c2fbf63cefa67cf8e8f681246f, ASSIGN 2023-02-13 21:58:07,111 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ef75de132134b515dbe11abe569ac292, ASSIGN 2023-02-13 21:58:07,112 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:58:07,112 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=00d462c2fbf63cefa67cf8e8f681246f, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,39523,1676325451480; forceNewPlan=false, retain=true 2023-02-13 21:58:07,112 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:58:07,112 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ef75de132134b515dbe11abe569ac292, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,36339,1676325451407; forceNewPlan=false, retain=true 2023-02-13 21:58:07,234 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 21:58:07,261 INFO [jenkins-hbase20:46557] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 21:58:07,261 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=00d462c2fbf63cefa67cf8e8f681246f, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:58:07,261 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=ac1d4ae96948f4de74aa3114b7308487, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:58:07,261 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=f244b6e82e7a1752e9a3cb5f28dea2e9, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:58:07,261 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676325487261"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325487261"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325487261"}]},"ts":"1676325487261"} 2023-02-13 21:58:07,261 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=1c996d2a582a7ec43382d5a39bb54c78, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:58:07,261 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=d1acdd49740dd716d5bec22786474a40, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:58:07,262 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676325487261"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325487261"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325487261"}]},"ts":"1676325487261"} 2023-02-13 21:58:07,261 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325487261"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325487261"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325487261"}]},"ts":"1676325487261"} 2023-02-13 21:58:07,261 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325487261"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325487261"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325487261"}]},"ts":"1676325487261"} 2023-02-13 21:58:07,262 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325487261"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325487261"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325487261"}]},"ts":"1676325487261"} 2023-02-13 21:58:07,263 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=263, ppid=262, state=RUNNABLE; OpenRegionProcedure 00d462c2fbf63cefa67cf8e8f681246f, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:58:07,263 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=264, ppid=257, state=RUNNABLE; OpenRegionProcedure 1c996d2a582a7ec43382d5a39bb54c78, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:58:07,264 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=265, ppid=258, state=RUNNABLE; OpenRegionProcedure f244b6e82e7a1752e9a3cb5f28dea2e9, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:58:07,264 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=266, ppid=259, state=RUNNABLE; OpenRegionProcedure ac1d4ae96948f4de74aa3114b7308487, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:58:07,265 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=267, ppid=260, state=RUNNABLE; OpenRegionProcedure d1acdd49740dd716d5bec22786474a40, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:58:07,266 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=ef75de132134b515dbe11abe569ac292, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:58:07,266 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325487266"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325487266"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325487266"}]},"ts":"1676325487266"} 2023-02-13 21:58:07,267 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=268, ppid=261, state=RUNNABLE; OpenRegionProcedure ef75de132134b515dbe11abe569ac292, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:58:07,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-13 21:58:07,417 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. 2023-02-13 21:58:07,417 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 00d462c2fbf63cefa67cf8e8f681246f, NAME => 'testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 21:58:07,417 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676325482126 00d462c2fbf63cefa67cf8e8f681246f 2023-02-13 21:58:07,417 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:58:07,417 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 00d462c2fbf63cefa67cf8e8f681246f 2023-02-13 21:58:07,417 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. 2023-02-13 21:58:07,417 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 00d462c2fbf63cefa67cf8e8f681246f 2023-02-13 21:58:07,417 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ef75de132134b515dbe11abe569ac292, NAME => 'testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 21:58:07,417 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. 2023-02-13 21:58:07,417 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676325482126 ef75de132134b515dbe11abe569ac292 2023-02-13 21:58:07,417 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f244b6e82e7a1752e9a3cb5f28dea2e9, NAME => 'testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 21:58:07,417 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:58:07,418 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for ef75de132134b515dbe11abe569ac292 2023-02-13 21:58:07,418 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676325482126 f244b6e82e7a1752e9a3cb5f28dea2e9 2023-02-13 21:58:07,418 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for ef75de132134b515dbe11abe569ac292 2023-02-13 21:58:07,418 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:58:07,418 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for f244b6e82e7a1752e9a3cb5f28dea2e9 2023-02-13 21:58:07,418 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for f244b6e82e7a1752e9a3cb5f28dea2e9 2023-02-13 21:58:07,418 INFO [StoreOpener-00d462c2fbf63cefa67cf8e8f681246f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 00d462c2fbf63cefa67cf8e8f681246f 2023-02-13 21:58:07,419 INFO [StoreOpener-f244b6e82e7a1752e9a3cb5f28dea2e9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f244b6e82e7a1752e9a3cb5f28dea2e9 2023-02-13 21:58:07,419 INFO [StoreOpener-ef75de132134b515dbe11abe569ac292-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ef75de132134b515dbe11abe569ac292 2023-02-13 21:58:07,419 DEBUG [StoreOpener-00d462c2fbf63cefa67cf8e8f681246f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/cf 2023-02-13 21:58:07,419 DEBUG [StoreOpener-00d462c2fbf63cefa67cf8e8f681246f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/cf 2023-02-13 21:58:07,420 INFO [StoreOpener-00d462c2fbf63cefa67cf8e8f681246f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 00d462c2fbf63cefa67cf8e8f681246f columnFamilyName cf 2023-02-13 21:58:07,420 DEBUG [StoreOpener-f244b6e82e7a1752e9a3cb5f28dea2e9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9/cf 2023-02-13 21:58:07,420 DEBUG [StoreOpener-f244b6e82e7a1752e9a3cb5f28dea2e9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9/cf 2023-02-13 21:58:07,420 DEBUG [StoreOpener-ef75de132134b515dbe11abe569ac292-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292/cf 2023-02-13 21:58:07,420 DEBUG [StoreOpener-ef75de132134b515dbe11abe569ac292-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292/cf 2023-02-13 21:58:07,420 INFO [StoreOpener-f244b6e82e7a1752e9a3cb5f28dea2e9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f244b6e82e7a1752e9a3cb5f28dea2e9 columnFamilyName cf 2023-02-13 21:58:07,420 INFO [StoreOpener-ef75de132134b515dbe11abe569ac292-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ef75de132134b515dbe11abe569ac292 columnFamilyName cf 2023-02-13 21:58:07,428 DEBUG [StoreOpener-00d462c2fbf63cefa67cf8e8f681246f-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/cf/ead9f71bf7144e80bc30d97efc70516f 2023-02-13 21:58:07,428 DEBUG [StoreOpener-f244b6e82e7a1752e9a3cb5f28dea2e9-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9/cf/b25fb604e9074f249714afdbfac4c6d3 2023-02-13 21:58:07,429 INFO [StoreOpener-00d462c2fbf63cefa67cf8e8f681246f-1] regionserver.HStore(310): Store=00d462c2fbf63cefa67cf8e8f681246f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:58:07,429 INFO [StoreOpener-f244b6e82e7a1752e9a3cb5f28dea2e9-1] regionserver.HStore(310): Store=f244b6e82e7a1752e9a3cb5f28dea2e9/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:58:07,429 DEBUG [StoreOpener-ef75de132134b515dbe11abe569ac292-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292/cf/8e37f961530e4488a37efc85ab1c86ac 2023-02-13 21:58:07,429 INFO [StoreOpener-ef75de132134b515dbe11abe569ac292-1] regionserver.HStore(310): Store=ef75de132134b515dbe11abe569ac292/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:58:07,429 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f 2023-02-13 21:58:07,429 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9 2023-02-13 21:58:07,429 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292 2023-02-13 21:58:07,430 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f 2023-02-13 21:58:07,430 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9 2023-02-13 21:58:07,430 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292 2023-02-13 21:58:07,432 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for f244b6e82e7a1752e9a3cb5f28dea2e9 2023-02-13 21:58:07,432 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 00d462c2fbf63cefa67cf8e8f681246f 2023-02-13 21:58:07,432 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for ef75de132134b515dbe11abe569ac292 2023-02-13 21:58:07,432 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 00d462c2fbf63cefa67cf8e8f681246f; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70832420, jitterRate=0.055485308170318604}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:58:07,432 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened f244b6e82e7a1752e9a3cb5f28dea2e9; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63948476, jitterRate=-0.04709345102310181}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:58:07,432 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened ef75de132134b515dbe11abe569ac292; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60643117, jitterRate=-0.0963471382856369}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:58:07,432 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for f244b6e82e7a1752e9a3cb5f28dea2e9: Running coprocessor pre-open hook at 1676325487418Writing region info on filesystem at 1676325487418Initializing all the Stores at 1676325487419 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325487419Cleaning up temporary data from old regions at 1676325487430 (+11 ms)Cleaning up detritus from prior splits at 1676325487431 (+1 ms)Running coprocessor post-open hooks at 1676325487432 (+1 ms)Region opened successfully at 1676325487432 2023-02-13 21:58:07,432 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 00d462c2fbf63cefa67cf8e8f681246f: Running coprocessor pre-open hook at 1676325487417Writing region info on filesystem at 1676325487417Initializing all the Stores at 1676325487418 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325487418Cleaning up temporary data from old regions at 1676325487430 (+12 ms)Cleaning up detritus from prior splits at 1676325487431 (+1 ms)Running coprocessor post-open hooks at 1676325487432 (+1 ms)Region opened successfully at 1676325487432 2023-02-13 21:58:07,432 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for ef75de132134b515dbe11abe569ac292: Running coprocessor pre-open hook at 1676325487418Writing region info on filesystem at 1676325487418Initializing all the Stores at 1676325487419 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325487419Cleaning up temporary data from old regions at 1676325487430 (+11 ms)Cleaning up detritus from prior splits at 1676325487431 (+1 ms)Running coprocessor post-open hooks at 1676325487432 (+1 ms)Region opened successfully at 1676325487432 2023-02-13 21:58:07,433 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292., pid=268, masterSystemTime=1676325487416 2023-02-13 21:58:07,433 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f., pid=263, masterSystemTime=1676325487414 2023-02-13 21:58:07,433 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9., pid=265, masterSystemTime=1676325487416 2023-02-13 21:58:07,434 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. 2023-02-13 21:58:07,434 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. 2023-02-13 21:58:07,434 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. 2023-02-13 21:58:07,434 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d1acdd49740dd716d5bec22786474a40, NAME => 'testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 21:58:07,434 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676325482126 d1acdd49740dd716d5bec22786474a40 2023-02-13 21:58:07,434 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:58:07,434 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for d1acdd49740dd716d5bec22786474a40 2023-02-13 21:58:07,434 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for d1acdd49740dd716d5bec22786474a40 2023-02-13 21:58:07,434 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=f244b6e82e7a1752e9a3cb5f28dea2e9, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:58:07,434 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. 2023-02-13 21:58:07,435 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325487434"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325487434"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325487434"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325487434"}]},"ts":"1676325487434"} 2023-02-13 21:58:07,435 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. 2023-02-13 21:58:07,435 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. 2023-02-13 21:58:07,435 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1c996d2a582a7ec43382d5a39bb54c78, NAME => 'testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 21:58:07,435 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=00d462c2fbf63cefa67cf8e8f681246f, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:58:07,435 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676325482126 1c996d2a582a7ec43382d5a39bb54c78 2023-02-13 21:58:07,435 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676325487435"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325487435"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325487435"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325487435"}]},"ts":"1676325487435"} 2023-02-13 21:58:07,435 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. 2023-02-13 21:58:07,435 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. 2023-02-13 21:58:07,435 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. 2023-02-13 21:58:07,435 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ac1d4ae96948f4de74aa3114b7308487, NAME => 'testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 21:58:07,435 INFO [StoreOpener-d1acdd49740dd716d5bec22786474a40-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d1acdd49740dd716d5bec22786474a40 2023-02-13 21:58:07,436 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676325482126 ac1d4ae96948f4de74aa3114b7308487 2023-02-13 21:58:07,436 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=ef75de132134b515dbe11abe569ac292, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:58:07,435 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:58:07,436 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 21:58:07,436 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325487436"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325487436"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325487436"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325487436"}]},"ts":"1676325487436"} 2023-02-13 21:58:07,436 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for ac1d4ae96948f4de74aa3114b7308487 2023-02-13 21:58:07,436 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for ac1d4ae96948f4de74aa3114b7308487 2023-02-13 21:58:07,436 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 1c996d2a582a7ec43382d5a39bb54c78 2023-02-13 21:58:07,436 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 1c996d2a582a7ec43382d5a39bb54c78 2023-02-13 21:58:07,436 DEBUG [StoreOpener-d1acdd49740dd716d5bec22786474a40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40/cf 2023-02-13 21:58:07,436 DEBUG [StoreOpener-d1acdd49740dd716d5bec22786474a40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40/cf 2023-02-13 21:58:07,437 INFO [StoreOpener-d1acdd49740dd716d5bec22786474a40-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d1acdd49740dd716d5bec22786474a40 columnFamilyName cf 2023-02-13 21:58:07,437 INFO [StoreOpener-ac1d4ae96948f4de74aa3114b7308487-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ac1d4ae96948f4de74aa3114b7308487 2023-02-13 21:58:07,437 INFO [StoreOpener-1c996d2a582a7ec43382d5a39bb54c78-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1c996d2a582a7ec43382d5a39bb54c78 2023-02-13 21:58:07,438 DEBUG [StoreOpener-ac1d4ae96948f4de74aa3114b7308487-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487/cf 2023-02-13 21:58:07,438 DEBUG [StoreOpener-ac1d4ae96948f4de74aa3114b7308487-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487/cf 2023-02-13 21:58:07,438 DEBUG [StoreOpener-1c996d2a582a7ec43382d5a39bb54c78-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78/cf 2023-02-13 21:58:07,438 DEBUG [StoreOpener-1c996d2a582a7ec43382d5a39bb54c78-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78/cf 2023-02-13 21:58:07,438 INFO [StoreOpener-ac1d4ae96948f4de74aa3114b7308487-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ac1d4ae96948f4de74aa3114b7308487 columnFamilyName cf 2023-02-13 21:58:07,438 INFO [StoreOpener-1c996d2a582a7ec43382d5a39bb54c78-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1c996d2a582a7ec43382d5a39bb54c78 columnFamilyName cf 2023-02-13 21:58:07,440 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=263, resume processing ppid=262 2023-02-13 21:58:07,440 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=265, resume processing ppid=258 2023-02-13 21:58:07,440 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=263, ppid=262, state=SUCCESS; OpenRegionProcedure 00d462c2fbf63cefa67cf8e8f681246f, server=jenkins-hbase20.apache.org,39523,1676325451480 in 174 msec 2023-02-13 21:58:07,440 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=268, resume processing ppid=261 2023-02-13 21:58:07,440 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=265, ppid=258, state=SUCCESS; OpenRegionProcedure f244b6e82e7a1752e9a3cb5f28dea2e9, server=jenkins-hbase20.apache.org,45015,1676325451448 in 172 msec 2023-02-13 21:58:07,440 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=268, ppid=261, state=SUCCESS; OpenRegionProcedure ef75de132134b515dbe11abe569ac292, server=jenkins-hbase20.apache.org,36339,1676325451407 in 170 msec 2023-02-13 21:58:07,441 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=261, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ef75de132134b515dbe11abe569ac292, ASSIGN in 331 msec 2023-02-13 21:58:07,441 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=258, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=f244b6e82e7a1752e9a3cb5f28dea2e9, ASSIGN in 331 msec 2023-02-13 21:58:07,442 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=262, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=00d462c2fbf63cefa67cf8e8f681246f, ASSIGN in 331 msec 2023-02-13 21:58:07,443 DEBUG [StoreOpener-d1acdd49740dd716d5bec22786474a40-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40/cf/278f4c4405a04414a1eb5319d07c2091 2023-02-13 21:58:07,444 INFO [StoreOpener-d1acdd49740dd716d5bec22786474a40-1] regionserver.HStore(310): Store=d1acdd49740dd716d5bec22786474a40/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:58:07,444 DEBUG [StoreOpener-ac1d4ae96948f4de74aa3114b7308487-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487/cf/a015f28c44be4a2f913a58fe70ca099e 2023-02-13 21:58:07,444 INFO [StoreOpener-ac1d4ae96948f4de74aa3114b7308487-1] regionserver.HStore(310): Store=ac1d4ae96948f4de74aa3114b7308487/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:58:07,444 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40 2023-02-13 21:58:07,445 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487 2023-02-13 21:58:07,445 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40 2023-02-13 21:58:07,445 DEBUG [StoreOpener-1c996d2a582a7ec43382d5a39bb54c78-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78/cf/f68183f886e84213becb66fd697bf854 2023-02-13 21:58:07,445 INFO [StoreOpener-1c996d2a582a7ec43382d5a39bb54c78-1] regionserver.HStore(310): Store=1c996d2a582a7ec43382d5a39bb54c78/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 21:58:07,445 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487 2023-02-13 21:58:07,446 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78 2023-02-13 21:58:07,446 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78 2023-02-13 21:58:07,446 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for d1acdd49740dd716d5bec22786474a40 2023-02-13 21:58:07,447 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for ac1d4ae96948f4de74aa3114b7308487 2023-02-13 21:58:07,447 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened d1acdd49740dd716d5bec22786474a40; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61539893, jitterRate=-0.08298413455486298}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:58:07,447 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for d1acdd49740dd716d5bec22786474a40: Running coprocessor pre-open hook at 1676325487435Writing region info on filesystem at 1676325487435Initializing all the Stores at 1676325487435Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325487435Cleaning up temporary data from old regions at 1676325487445 (+10 ms)Cleaning up detritus from prior splits at 1676325487445Running coprocessor post-open hooks at 1676325487447 (+2 ms)Region opened successfully at 1676325487447 2023-02-13 21:58:07,447 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened ac1d4ae96948f4de74aa3114b7308487; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64541303, jitterRate=-0.03825964033603668}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:58:07,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for ac1d4ae96948f4de74aa3114b7308487: Running coprocessor pre-open hook at 1676325487436Writing region info on filesystem at 1676325487436Initializing all the Stores at 1676325487437 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325487437Cleaning up temporary data from old regions at 1676325487446 (+9 ms)Cleaning up detritus from prior splits at 1676325487446Running coprocessor post-open hooks at 1676325487448 (+2 ms)Region opened successfully at 1676325487448 2023-02-13 21:58:07,448 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40., pid=267, masterSystemTime=1676325487416 2023-02-13 21:58:07,448 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487., pid=266, masterSystemTime=1676325487416 2023-02-13 21:58:07,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 1c996d2a582a7ec43382d5a39bb54c78 2023-02-13 21:58:07,449 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 1c996d2a582a7ec43382d5a39bb54c78; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65120206, jitterRate=-0.029633313417434692}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 21:58:07,449 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. 2023-02-13 21:58:07,449 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 1c996d2a582a7ec43382d5a39bb54c78: Running coprocessor pre-open hook at 1676325487436Writing region info on filesystem at 1676325487436Initializing all the Stores at 1676325487437 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676325487437Cleaning up temporary data from old regions at 1676325487447 (+10 ms)Cleaning up detritus from prior splits at 1676325487447Running coprocessor post-open hooks at 1676325487449 (+2 ms)Region opened successfully at 1676325487449 2023-02-13 21:58:07,449 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. 2023-02-13 21:58:07,449 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=d1acdd49740dd716d5bec22786474a40, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:58:07,449 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. 2023-02-13 21:58:07,449 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. 2023-02-13 21:58:07,449 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325487449"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325487449"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325487449"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325487449"}]},"ts":"1676325487449"} 2023-02-13 21:58:07,449 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78., pid=264, masterSystemTime=1676325487414 2023-02-13 21:58:07,450 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=ac1d4ae96948f4de74aa3114b7308487, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:58:07,450 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325487450"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325487450"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325487450"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325487450"}]},"ts":"1676325487450"} 2023-02-13 21:58:07,451 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. 2023-02-13 21:58:07,451 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. 2023-02-13 21:58:07,451 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=1c996d2a582a7ec43382d5a39bb54c78, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:58:07,451 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676325487451"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676325487451"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676325487451"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676325487451"}]},"ts":"1676325487451"} 2023-02-13 21:58:07,452 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=267, resume processing ppid=260 2023-02-13 21:58:07,452 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=267, ppid=260, state=SUCCESS; OpenRegionProcedure d1acdd49740dd716d5bec22786474a40, server=jenkins-hbase20.apache.org,45015,1676325451448 in 186 msec 2023-02-13 21:58:07,452 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=266, resume processing ppid=259 2023-02-13 21:58:07,452 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=266, ppid=259, state=SUCCESS; OpenRegionProcedure ac1d4ae96948f4de74aa3114b7308487, server=jenkins-hbase20.apache.org,36339,1676325451407 in 187 msec 2023-02-13 21:58:07,452 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=260, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=d1acdd49740dd716d5bec22786474a40, ASSIGN in 343 msec 2023-02-13 21:58:07,453 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=259, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ac1d4ae96948f4de74aa3114b7308487, ASSIGN in 343 msec 2023-02-13 21:58:07,453 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=264, resume processing ppid=257 2023-02-13 21:58:07,453 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=264, ppid=257, state=SUCCESS; OpenRegionProcedure 1c996d2a582a7ec43382d5a39bb54c78, server=jenkins-hbase20.apache.org,39523,1676325451480 in 189 msec 2023-02-13 21:58:07,454 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=257, resume processing ppid=256 2023-02-13 21:58:07,454 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=257, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=1c996d2a582a7ec43382d5a39bb54c78, ASSIGN in 344 msec 2023-02-13 21:58:07,454 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676325482126","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325487454"}]},"ts":"1676325487454"} 2023-02-13 21:58:07,455 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676325482126, state=ENABLED in hbase:meta 2023-02-13 21:58:07,456 INFO [PEWorker-2] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1676325482126' was successfully enabled. 2023-02-13 21:58:07,457 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=256, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1676325482126 in 357 msec 2023-02-13 21:58:07,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-13 21:58:07,856 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1676325482126, procId: 256 completed 2023-02-13 21:58:07,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 21:58:07,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 21:58:07,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36339] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 21:58:07,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45015] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 21:58:07,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 21:58:07,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39523] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 21:58:07,872 DEBUG [Listener at localhost.localdomain/44075] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676325482126 2023-02-13 21:58:07,872 DEBUG [Listener at localhost.localdomain/44075] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. 2023-02-13 21:58:07,872 INFO [Listener at localhost.localdomain/44075] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 21:58:07,981 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 21:58:08,099 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 21:58:08,201 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 21:58:08,294 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 21:58:08,402 DEBUG [Listener at localhost.localdomain/44075] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 21:58:10,045 ERROR [VerifySnapshot-pool-0] snapshot.SnapshotReferenceUtil(301): Can't find hfile: ead9f71bf7144e80bc30d97efc70516f in the real (hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/cf/ead9f71bf7144e80bc30d97efc70516f) or archive (hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/cf/ead9f71bf7144e80bc30d97efc70516f) directory for the primary table. 2023-02-13 21:58:10,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: ead9f71bf7144e80bc30d97efc70516f in the real (hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/cf/ead9f71bf7144e80bc30d97efc70516f) or archive (hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/cf/ead9f71bf7144e80bc30d97efc70516f) directory for the primary table. at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:225) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:187) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.verifySnapshot(SnapshotReferenceUtil.java:164) at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.restoreOrCloneSnapshot(SnapshotManager.java:827) at org.apache.hadoop.hbase.master.HMaster$14.run(HMaster.java:2493) at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:132) at org.apache.hadoop.hbase.master.HMaster.restoreSnapshot(HMaster.java:2489) at org.apache.hadoop.hbase.master.MasterRpcServices.restoreSnapshot(MasterRpcServices.java:1469) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 21:58:10,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] ipc.CallRunner(144): callId: 366 service: MasterService methodName: RestoreSnapshot size: 114 connection: 148.251.75.209:57230 deadline: 1676325550028, exception=org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: ead9f71bf7144e80bc30d97efc70516f in the real (hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/cf/ead9f71bf7144e80bc30d97efc70516f) or archive (hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/cf/ead9f71bf7144e80bc30d97efc70516f) directory for the primary table. 2023-02-13 21:58:10,052 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676325482126 2023-02-13 21:58:10,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.HMaster$11(2405): Client=jenkins//148.251.75.209 disable testCorruptedSnapshot-1676325482126 2023-02-13 21:58:10,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=270, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676325482126 2023-02-13 21:58:10,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-13 21:58:10,058 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676325482126","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325490058"}]},"ts":"1676325490058"} 2023-02-13 21:58:10,059 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676325482126, state=DISABLING in hbase:meta 2023-02-13 21:58:10,061 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676325482126 to state=DISABLING 2023-02-13 21:58:10,062 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=1c996d2a582a7ec43382d5a39bb54c78, UNASSIGN}, {pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=f244b6e82e7a1752e9a3cb5f28dea2e9, UNASSIGN}, {pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ac1d4ae96948f4de74aa3114b7308487, UNASSIGN}, {pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=d1acdd49740dd716d5bec22786474a40, UNASSIGN}, {pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ef75de132134b515dbe11abe569ac292, UNASSIGN}, {pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=00d462c2fbf63cefa67cf8e8f681246f, UNASSIGN}] 2023-02-13 21:58:10,062 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=f244b6e82e7a1752e9a3cb5f28dea2e9, UNASSIGN 2023-02-13 21:58:10,062 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ac1d4ae96948f4de74aa3114b7308487, UNASSIGN 2023-02-13 21:58:10,062 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=1c996d2a582a7ec43382d5a39bb54c78, UNASSIGN 2023-02-13 21:58:10,063 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=d1acdd49740dd716d5bec22786474a40, UNASSIGN 2023-02-13 21:58:10,063 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=00d462c2fbf63cefa67cf8e8f681246f, UNASSIGN 2023-02-13 21:58:10,063 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=f244b6e82e7a1752e9a3cb5f28dea2e9, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:58:10,063 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=1c996d2a582a7ec43382d5a39bb54c78, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:58:10,064 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325490063"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325490063"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325490063"}]},"ts":"1676325490063"} 2023-02-13 21:58:10,064 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676325490063"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325490063"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325490063"}]},"ts":"1676325490063"} 2023-02-13 21:58:10,064 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=ac1d4ae96948f4de74aa3114b7308487, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:58:10,064 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=d1acdd49740dd716d5bec22786474a40, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:58:10,064 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325490064"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325490064"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325490064"}]},"ts":"1676325490064"} 2023-02-13 21:58:10,064 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325490064"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325490064"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325490064"}]},"ts":"1676325490064"} 2023-02-13 21:58:10,064 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=00d462c2fbf63cefa67cf8e8f681246f, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:58:10,064 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676325490064"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325490064"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325490064"}]},"ts":"1676325490064"} 2023-02-13 21:58:10,065 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=277, ppid=272, state=RUNNABLE; CloseRegionProcedure f244b6e82e7a1752e9a3cb5f28dea2e9, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:58:10,066 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=278, ppid=271, state=RUNNABLE; CloseRegionProcedure 1c996d2a582a7ec43382d5a39bb54c78, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:58:10,067 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=279, ppid=273, state=RUNNABLE; CloseRegionProcedure ac1d4ae96948f4de74aa3114b7308487, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:58:10,068 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=280, ppid=274, state=RUNNABLE; CloseRegionProcedure d1acdd49740dd716d5bec22786474a40, server=jenkins-hbase20.apache.org,45015,1676325451448}] 2023-02-13 21:58:10,069 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=281, ppid=276, state=RUNNABLE; CloseRegionProcedure 00d462c2fbf63cefa67cf8e8f681246f, server=jenkins-hbase20.apache.org,39523,1676325451480}] 2023-02-13 21:58:10,071 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ef75de132134b515dbe11abe569ac292, UNASSIGN 2023-02-13 21:58:10,072 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=ef75de132134b515dbe11abe569ac292, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:58:10,072 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325490072"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676325490072"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676325490072"}]},"ts":"1676325490072"} 2023-02-13 21:58:10,074 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=282, ppid=275, state=RUNNABLE; CloseRegionProcedure ef75de132134b515dbe11abe569ac292, server=jenkins-hbase20.apache.org,36339,1676325451407}] 2023-02-13 21:58:10,219 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close d1acdd49740dd716d5bec22786474a40 2023-02-13 21:58:10,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing d1acdd49740dd716d5bec22786474a40, disabling compactions & flushes 2023-02-13 21:58:10,219 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. 2023-02-13 21:58:10,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. 2023-02-13 21:58:10,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. after waiting 0 ms 2023-02-13 21:58:10,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. 2023-02-13 21:58:10,220 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing d1acdd49740dd716d5bec22786474a40 1/1 column families, dataSize=1.90 KB heapSize=4.33 KB 2023-02-13 21:58:10,220 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 00d462c2fbf63cefa67cf8e8f681246f 2023-02-13 21:58:10,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 00d462c2fbf63cefa67cf8e8f681246f, disabling compactions & flushes 2023-02-13 21:58:10,220 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. 2023-02-13 21:58:10,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. 2023-02-13 21:58:10,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. after waiting 0 ms 2023-02-13 21:58:10,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. 2023-02-13 21:58:10,221 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing 00d462c2fbf63cefa67cf8e8f681246f 1/1 column families, dataSize=21.98 KB heapSize=47.50 KB 2023-02-13 21:58:10,223 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close ef75de132134b515dbe11abe569ac292 2023-02-13 21:58:10,223 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing ef75de132134b515dbe11abe569ac292, disabling compactions & flushes 2023-02-13 21:58:10,223 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. 2023-02-13 21:58:10,224 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. 2023-02-13 21:58:10,224 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. after waiting 0 ms 2023-02-13 21:58:10,224 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. 2023-02-13 21:58:10,224 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing ef75de132134b515dbe11abe569ac292 1/1 column families, dataSize=1.63 KB heapSize=3.77 KB 2023-02-13 21:58:10,241 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230213455659ab23484199a9c928783d9387f6 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213455659ab23484199a9c928783d9387f6 2023-02-13 21:58:10,241 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d5202302135a2601c9b72e49aaa036c357b67807bf to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/e4da3b7fbbce2345d7772b0674a318d5202302135a2601c9b72e49aaa036c357b67807bf 2023-02-13 21:58:10,241 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c202302130484dfc9fc0b4b8790133632a55a501e to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/a87ff679a2f3e71d9181a67b7542122c202302130484dfc9fc0b4b8790133632a55a501e 2023-02-13 21:58:10,247 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40/.tmp/cf/26b9577a905343918c9c51119195cf7c 2023-02-13 21:58:10,248 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=22.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/.tmp/cf/263d4d83a7ee44d9a8f0df3d680818c6 2023-02-13 21:58:10,253 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40/.tmp/cf/26b9577a905343918c9c51119195cf7c as hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40/cf/26b9577a905343918c9c51119195cf7c 2023-02-13 21:58:10,253 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/.tmp/cf/263d4d83a7ee44d9a8f0df3d680818c6 as hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/cf/263d4d83a7ee44d9a8f0df3d680818c6 2023-02-13 21:58:10,256 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40/cf/26b9577a905343918c9c51119195cf7c, entries=29, sequenceid=15, filesize=9.9 K 2023-02-13 21:58:10,256 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/cf/263d4d83a7ee44d9a8f0df3d680818c6, entries=336, sequenceid=15, filesize=62.8 K 2023-02-13 21:58:10,256 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for d1acdd49740dd716d5bec22786474a40 in 37ms, sequenceid=15, compaction requested=false 2023-02-13 21:58:10,258 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~21.98 KB/22508, heapSize ~47.48 KB/48624, currentSize=0 B/0 for 00d462c2fbf63cefa67cf8e8f681246f in 38ms, sequenceid=15, compaction requested=false 2023-02-13 21:58:10,261 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 21:58:10,261 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 21:58:10,261 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. 2023-02-13 21:58:10,261 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. 2023-02-13 21:58:10,261 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 00d462c2fbf63cefa67cf8e8f681246f: Waiting for close lock at 1676325490220Running coprocessor pre-close hooks at 1676325490220Disabling compacts and flushes for region at 1676325490220Disabling writes for close at 1676325490220Obtaining lock to block concurrent updates at 1676325490222 (+2 ms)Preparing flush snapshotting stores in 00d462c2fbf63cefa67cf8e8f681246f at 1676325490222Finished memstore snapshotting testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f., syncing WAL and waiting on mvcc, flushsize=dataSize=22508, getHeapSize=48624, getOffHeapSize=0, getCellsCount=336 at 1676325490222Flushing stores of testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f. at 1676325490223 (+1 ms)Flushing 00d462c2fbf63cefa67cf8e8f681246f/cf: creating writer at 1676325490224 (+1 ms)Flushing mob file 00d462c2fbf63cefa67cf8e8f681246f/cf: appending metadata at 1676325490232 (+8 ms)Flushing mob file 00d462c2fbf63cefa67cf8e8f681246f/cf: closing flushed file at 1676325490232Flushing 00d462c2fbf63cefa67cf8e8f681246f/cf: appending metadata at 1676325490242 (+10 ms)Flushing 00d462c2fbf63cefa67cf8e8f681246f/cf: closing flushed file at 1676325490242Flushing 00d462c2fbf63cefa67cf8e8f681246f/cf: reopening flushed file at 1676325490254 (+12 ms)Finished flush of dataSize ~21.98 KB/22508, heapSize ~47.48 KB/48624, currentSize=0 B/0 for 00d462c2fbf63cefa67cf8e8f681246f in 38ms, sequenceid=15, compaction requested=false at 1676325490258 (+4 ms)Writing region close event to WAL at 1676325490259 (+1 ms)Running coprocessor post-close hooks at 1676325490261 (+2 ms)Closed at 1676325490261 2023-02-13 21:58:10,261 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for d1acdd49740dd716d5bec22786474a40: Waiting for close lock at 1676325490219Running coprocessor pre-close hooks at 1676325490219Disabling compacts and flushes for region at 1676325490219Disabling writes for close at 1676325490219Obtaining lock to block concurrent updates at 1676325490220 (+1 ms)Preparing flush snapshotting stores in d1acdd49740dd716d5bec22786474a40 at 1676325490220Finished memstore snapshotting testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40., syncing WAL and waiting on mvcc, flushsize=dataSize=1941, getHeapSize=4416, getOffHeapSize=0, getCellsCount=29 at 1676325490220Flushing stores of testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40. at 1676325490221 (+1 ms)Flushing d1acdd49740dd716d5bec22786474a40/cf: creating writer at 1676325490222 (+1 ms)Flushing mob file d1acdd49740dd716d5bec22786474a40/cf: appending metadata at 1676325490230 (+8 ms)Flushing mob file d1acdd49740dd716d5bec22786474a40/cf: closing flushed file at 1676325490230Flushing d1acdd49740dd716d5bec22786474a40/cf: appending metadata at 1676325490242 (+12 ms)Flushing d1acdd49740dd716d5bec22786474a40/cf: closing flushed file at 1676325490242Flushing d1acdd49740dd716d5bec22786474a40/cf: reopening flushed file at 1676325490253 (+11 ms)Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for d1acdd49740dd716d5bec22786474a40 in 37ms, sequenceid=15, compaction requested=false at 1676325490256 (+3 ms)Writing region close event to WAL at 1676325490259 (+3 ms)Running coprocessor post-close hooks at 1676325490261 (+2 ms)Closed at 1676325490261 2023-02-13 21:58:10,262 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 00d462c2fbf63cefa67cf8e8f681246f 2023-02-13 21:58:10,263 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 1c996d2a582a7ec43382d5a39bb54c78 2023-02-13 21:58:10,263 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 1c996d2a582a7ec43382d5a39bb54c78, disabling compactions & flushes 2023-02-13 21:58:10,263 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. 2023-02-13 21:58:10,263 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. 2023-02-13 21:58:10,263 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. after waiting 0 ms 2023-02-13 21:58:10,263 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. 2023-02-13 21:58:10,263 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=00d462c2fbf63cefa67cf8e8f681246f, regionState=CLOSED 2023-02-13 21:58:10,263 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing 1c996d2a582a7ec43382d5a39bb54c78 1/1 column families, dataSize=2.62 KB heapSize=5.88 KB 2023-02-13 21:58:10,263 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676325490263"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325490263"}]},"ts":"1676325490263"} 2023-02-13 21:58:10,263 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed d1acdd49740dd716d5bec22786474a40 2023-02-13 21:58:10,263 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close f244b6e82e7a1752e9a3cb5f28dea2e9 2023-02-13 21:58:10,263 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing f244b6e82e7a1752e9a3cb5f28dea2e9, disabling compactions & flushes 2023-02-13 21:58:10,263 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. 2023-02-13 21:58:10,263 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. 2023-02-13 21:58:10,263 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=d1acdd49740dd716d5bec22786474a40, regionState=CLOSED 2023-02-13 21:58:10,263 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. after waiting 0 ms 2023-02-13 21:58:10,263 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325490263"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325490263"}]},"ts":"1676325490263"} 2023-02-13 21:58:10,263 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. 2023-02-13 21:58:10,263 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing f244b6e82e7a1752e9a3cb5f28dea2e9 1/1 column families, dataSize=2.48 KB heapSize=5.59 KB 2023-02-13 21:58:10,267 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=281, resume processing ppid=276 2023-02-13 21:58:10,267 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=281, ppid=276, state=SUCCESS; CloseRegionProcedure 00d462c2fbf63cefa67cf8e8f681246f, server=jenkins-hbase20.apache.org,39523,1676325451480 in 196 msec 2023-02-13 21:58:10,267 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=280, resume processing ppid=274 2023-02-13 21:58:10,267 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=280, ppid=274, state=SUCCESS; CloseRegionProcedure d1acdd49740dd716d5bec22786474a40, server=jenkins-hbase20.apache.org,45015,1676325451448 in 198 msec 2023-02-13 21:58:10,268 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=276, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=00d462c2fbf63cefa67cf8e8f681246f, UNASSIGN in 206 msec 2023-02-13 21:58:10,268 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=274, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=d1acdd49740dd716d5bec22786474a40, UNASSIGN in 206 msec 2023-02-13 21:58:10,277 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230213481947820bd64f349ecb71771348d25d to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/d41d8cd98f00b204e9800998ecf8427e20230213481947820bd64f349ecb71771348d25d 2023-02-13 21:58:10,277 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230213120a326c5afb489fa15b503a4197b66f to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/c4ca4238a0b923820dcc509a6f75849b20230213120a326c5afb489fa15b503a4197b66f 2023-02-13 21:58:10,285 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78/.tmp/cf/b4c0b7d99e734eb1bb2d11d74845fa68 2023-02-13 21:58:10,285 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9/.tmp/cf/aa2607acaf274ac8a033abba623fad77 2023-02-13 21:58:10,287 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78/.tmp/cf/b4c0b7d99e734eb1bb2d11d74845fa68 as hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78/cf/b4c0b7d99e734eb1bb2d11d74845fa68 2023-02-13 21:58:10,287 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9/.tmp/cf/aa2607acaf274ac8a033abba623fad77 as hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9/cf/aa2607acaf274ac8a033abba623fad77 2023-02-13 21:58:10,290 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78/cf/b4c0b7d99e734eb1bb2d11d74845fa68, entries=40, sequenceid=15, filesize=11.8 K 2023-02-13 21:58:10,290 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9/cf/aa2607acaf274ac8a033abba623fad77, entries=38, sequenceid=15, filesize=11.4 K 2023-02-13 21:58:10,291 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.62 KB/2678, heapSize ~5.86 KB/6000, currentSize=0 B/0 for 1c996d2a582a7ec43382d5a39bb54c78 in 27ms, sequenceid=15, compaction requested=false 2023-02-13 21:58:10,291 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.48 KB/2544, heapSize ~5.58 KB/5712, currentSize=0 B/0 for f244b6e82e7a1752e9a3cb5f28dea2e9 in 27ms, sequenceid=15, compaction requested=false 2023-02-13 21:58:10,293 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 21:58:10,293 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 21:58:10,293 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. 2023-02-13 21:58:10,293 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. 2023-02-13 21:58:10,293 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 1c996d2a582a7ec43382d5a39bb54c78: Waiting for close lock at 1676325490263Running coprocessor pre-close hooks at 1676325490263Disabling compacts and flushes for region at 1676325490263Disabling writes for close at 1676325490263Obtaining lock to block concurrent updates at 1676325490263Preparing flush snapshotting stores in 1c996d2a582a7ec43382d5a39bb54c78 at 1676325490263Finished memstore snapshotting testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78., syncing WAL and waiting on mvcc, flushsize=dataSize=2678, getHeapSize=6000, getOffHeapSize=0, getCellsCount=40 at 1676325490263Flushing stores of testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78. at 1676325490263Flushing 1c996d2a582a7ec43382d5a39bb54c78/cf: creating writer at 1676325490264 (+1 ms)Flushing mob file 1c996d2a582a7ec43382d5a39bb54c78/cf: appending metadata at 1676325490269 (+5 ms)Flushing mob file 1c996d2a582a7ec43382d5a39bb54c78/cf: closing flushed file at 1676325490269Flushing 1c996d2a582a7ec43382d5a39bb54c78/cf: appending metadata at 1676325490278 (+9 ms)Flushing 1c996d2a582a7ec43382d5a39bb54c78/cf: closing flushed file at 1676325490278Flushing 1c996d2a582a7ec43382d5a39bb54c78/cf: reopening flushed file at 1676325490288 (+10 ms)Finished flush of dataSize ~2.62 KB/2678, heapSize ~5.86 KB/6000, currentSize=0 B/0 for 1c996d2a582a7ec43382d5a39bb54c78 in 27ms, sequenceid=15, compaction requested=false at 1676325490291 (+3 ms)Writing region close event to WAL at 1676325490291Running coprocessor post-close hooks at 1676325490293 (+2 ms)Closed at 1676325490293 2023-02-13 21:58:10,293 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for f244b6e82e7a1752e9a3cb5f28dea2e9: Waiting for close lock at 1676325490263Running coprocessor pre-close hooks at 1676325490263Disabling compacts and flushes for region at 1676325490263Disabling writes for close at 1676325490263Obtaining lock to block concurrent updates at 1676325490263Preparing flush snapshotting stores in f244b6e82e7a1752e9a3cb5f28dea2e9 at 1676325490263Finished memstore snapshotting testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9., syncing WAL and waiting on mvcc, flushsize=dataSize=2544, getHeapSize=5712, getOffHeapSize=0, getCellsCount=38 at 1676325490264 (+1 ms)Flushing stores of testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9. at 1676325490265 (+1 ms)Flushing f244b6e82e7a1752e9a3cb5f28dea2e9/cf: creating writer at 1676325490265Flushing mob file f244b6e82e7a1752e9a3cb5f28dea2e9/cf: appending metadata at 1676325490269 (+4 ms)Flushing mob file f244b6e82e7a1752e9a3cb5f28dea2e9/cf: closing flushed file at 1676325490269Flushing f244b6e82e7a1752e9a3cb5f28dea2e9/cf: appending metadata at 1676325490278 (+9 ms)Flushing f244b6e82e7a1752e9a3cb5f28dea2e9/cf: closing flushed file at 1676325490278Flushing f244b6e82e7a1752e9a3cb5f28dea2e9/cf: reopening flushed file at 1676325490288 (+10 ms)Finished flush of dataSize ~2.48 KB/2544, heapSize ~5.58 KB/5712, currentSize=0 B/0 for f244b6e82e7a1752e9a3cb5f28dea2e9 in 27ms, sequenceid=15, compaction requested=false at 1676325490291 (+3 ms)Writing region close event to WAL at 1676325490291Running coprocessor post-close hooks at 1676325490293 (+2 ms)Closed at 1676325490293 2023-02-13 21:58:10,294 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 1c996d2a582a7ec43382d5a39bb54c78 2023-02-13 21:58:10,295 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=1c996d2a582a7ec43382d5a39bb54c78, regionState=CLOSED 2023-02-13 21:58:10,295 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed f244b6e82e7a1752e9a3cb5f28dea2e9 2023-02-13 21:58:10,295 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676325490295"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325490295"}]},"ts":"1676325490295"} 2023-02-13 21:58:10,295 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=f244b6e82e7a1752e9a3cb5f28dea2e9, regionState=CLOSED 2023-02-13 21:58:10,295 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325490295"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325490295"}]},"ts":"1676325490295"} 2023-02-13 21:58:10,297 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=278, resume processing ppid=271 2023-02-13 21:58:10,297 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=278, ppid=271, state=SUCCESS; CloseRegionProcedure 1c996d2a582a7ec43382d5a39bb54c78, server=jenkins-hbase20.apache.org,39523,1676325451480 in 230 msec 2023-02-13 21:58:10,297 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=277, resume processing ppid=272 2023-02-13 21:58:10,297 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=277, ppid=272, state=SUCCESS; CloseRegionProcedure f244b6e82e7a1752e9a3cb5f28dea2e9, server=jenkins-hbase20.apache.org,45015,1676325451448 in 231 msec 2023-02-13 21:58:10,297 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=271, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=1c996d2a582a7ec43382d5a39bb54c78, UNASSIGN in 236 msec 2023-02-13 21:58:10,298 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=272, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=f244b6e82e7a1752e9a3cb5f28dea2e9, UNASSIGN in 236 msec 2023-02-13 21:58:10,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-13 21:58:10,651 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292/.tmp/cf/6dc812778aed4605b9ff3224823d6d0f 2023-02-13 21:58:10,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292/.tmp/cf/6dc812778aed4605b9ff3224823d6d0f as hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292/cf/6dc812778aed4605b9ff3224823d6d0f 2023-02-13 21:58:10,658 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292/cf/6dc812778aed4605b9ff3224823d6d0f, entries=25, sequenceid=15, filesize=9.2 K 2023-02-13 21:58:10,659 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.63 KB/1673, heapSize ~3.75 KB/3840, currentSize=0 B/0 for ef75de132134b515dbe11abe569ac292 in 435ms, sequenceid=15, compaction requested=false 2023-02-13 21:58:10,663 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 21:58:10,663 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. 2023-02-13 21:58:10,664 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for ef75de132134b515dbe11abe569ac292: Waiting for close lock at 1676325490223Running coprocessor pre-close hooks at 1676325490223Disabling compacts and flushes for region at 1676325490223Disabling writes for close at 1676325490224 (+1 ms)Obtaining lock to block concurrent updates at 1676325490225 (+1 ms)Preparing flush snapshotting stores in ef75de132134b515dbe11abe569ac292 at 1676325490225Finished memstore snapshotting testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292., syncing WAL and waiting on mvcc, flushsize=dataSize=1673, getHeapSize=3840, getOffHeapSize=0, getCellsCount=25 at 1676325490225Flushing stores of testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292. at 1676325490225Flushing ef75de132134b515dbe11abe569ac292/cf: creating writer at 1676325490227 (+2 ms)Flushing mob file ef75de132134b515dbe11abe569ac292/cf: appending metadata at 1676325490232 (+5 ms)Flushing mob file ef75de132134b515dbe11abe569ac292/cf: closing flushed file at 1676325490232Flushing ef75de132134b515dbe11abe569ac292/cf: appending metadata at 1676325490242 (+10 ms)Flushing ef75de132134b515dbe11abe569ac292/cf: closing flushed file at 1676325490242Flushing ef75de132134b515dbe11abe569ac292/cf: reopening flushed file at 1676325490656 (+414 ms)Finished flush of dataSize ~1.63 KB/1673, heapSize ~3.75 KB/3840, currentSize=0 B/0 for ef75de132134b515dbe11abe569ac292 in 435ms, sequenceid=15, compaction requested=false at 1676325490659 (+3 ms)Writing region close event to WAL at 1676325490660 (+1 ms)Running coprocessor post-close hooks at 1676325490663 (+3 ms)Closed at 1676325490663 2023-02-13 21:58:10,665 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed ef75de132134b515dbe11abe569ac292 2023-02-13 21:58:10,665 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close ac1d4ae96948f4de74aa3114b7308487 2023-02-13 21:58:10,665 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=ef75de132134b515dbe11abe569ac292, regionState=CLOSED 2023-02-13 21:58:10,665 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325490665"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325490665"}]},"ts":"1676325490665"} 2023-02-13 21:58:10,665 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing ac1d4ae96948f4de74aa3114b7308487, disabling compactions & flushes 2023-02-13 21:58:10,665 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. 2023-02-13 21:58:10,665 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. 2023-02-13 21:58:10,665 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. after waiting 0 ms 2023-02-13 21:58:10,665 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. 2023-02-13 21:58:10,666 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing ac1d4ae96948f4de74aa3114b7308487 1/1 column families, dataSize=2.09 KB heapSize=4.75 KB 2023-02-13 21:58:10,669 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=282, resume processing ppid=275 2023-02-13 21:58:10,669 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=282, ppid=275, state=SUCCESS; CloseRegionProcedure ef75de132134b515dbe11abe569ac292, server=jenkins-hbase20.apache.org,36339,1676325451407 in 593 msec 2023-02-13 21:58:10,670 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=275, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ef75de132134b515dbe11abe569ac292, UNASSIGN in 608 msec 2023-02-13 21:58:10,694 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230213caf8e27114ce40648900d149e1450fb6 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/c81e728d9d4c2f636f067f89cc14862c20230213caf8e27114ce40648900d149e1450fb6 2023-02-13 21:58:10,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-13 21:58:11,108 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487/.tmp/cf/c70f1e666c604cf797b385efc5d7af97 2023-02-13 21:58:11,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487/.tmp/cf/c70f1e666c604cf797b385efc5d7af97 as hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487/cf/c70f1e666c604cf797b385efc5d7af97 2023-02-13 21:58:11,115 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487/cf/c70f1e666c604cf797b385efc5d7af97, entries=32, sequenceid=15, filesize=10.4 K 2023-02-13 21:58:11,116 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for ac1d4ae96948f4de74aa3114b7308487 in 451ms, sequenceid=15, compaction requested=false 2023-02-13 21:58:11,129 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 21:58:11,130 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. 2023-02-13 21:58:11,130 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for ac1d4ae96948f4de74aa3114b7308487: Waiting for close lock at 1676325490665Running coprocessor pre-close hooks at 1676325490665Disabling compacts and flushes for region at 1676325490665Disabling writes for close at 1676325490665Obtaining lock to block concurrent updates at 1676325490666 (+1 ms)Preparing flush snapshotting stores in ac1d4ae96948f4de74aa3114b7308487 at 1676325490666Finished memstore snapshotting testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487., syncing WAL and waiting on mvcc, flushsize=dataSize=2142, getHeapSize=4848, getOffHeapSize=0, getCellsCount=32 at 1676325490666Flushing stores of testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487. at 1676325490666Flushing ac1d4ae96948f4de74aa3114b7308487/cf: creating writer at 1676325490667 (+1 ms)Flushing mob file ac1d4ae96948f4de74aa3114b7308487/cf: appending metadata at 1676325490672 (+5 ms)Flushing mob file ac1d4ae96948f4de74aa3114b7308487/cf: closing flushed file at 1676325490672Flushing ac1d4ae96948f4de74aa3114b7308487/cf: appending metadata at 1676325490695 (+23 ms)Flushing ac1d4ae96948f4de74aa3114b7308487/cf: closing flushed file at 1676325490695Flushing ac1d4ae96948f4de74aa3114b7308487/cf: reopening flushed file at 1676325491113 (+418 ms)Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for ac1d4ae96948f4de74aa3114b7308487 in 451ms, sequenceid=15, compaction requested=false at 1676325491116 (+3 ms)Writing region close event to WAL at 1676325491117 (+1 ms)Running coprocessor post-close hooks at 1676325491130 (+13 ms)Closed at 1676325491130 2023-02-13 21:58:11,131 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed ac1d4ae96948f4de74aa3114b7308487 2023-02-13 21:58:11,131 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=ac1d4ae96948f4de74aa3114b7308487, regionState=CLOSED 2023-02-13 21:58:11,131 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676325491131"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676325491131"}]},"ts":"1676325491131"} 2023-02-13 21:58:11,133 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=279, resume processing ppid=273 2023-02-13 21:58:11,133 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=279, ppid=273, state=SUCCESS; CloseRegionProcedure ac1d4ae96948f4de74aa3114b7308487, server=jenkins-hbase20.apache.org,36339,1676325451407 in 1.0650 sec 2023-02-13 21:58:11,134 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=273, resume processing ppid=270 2023-02-13 21:58:11,134 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=273, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676325482126, region=ac1d4ae96948f4de74aa3114b7308487, UNASSIGN in 1.0720 sec 2023-02-13 21:58:11,135 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676325482126","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676325491135"}]},"ts":"1676325491135"} 2023-02-13 21:58:11,136 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676325482126, state=DISABLED in hbase:meta 2023-02-13 21:58:11,137 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676325482126 to state=DISABLED 2023-02-13 21:58:11,138 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=270, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676325482126 in 1.0850 sec 2023-02-13 21:58:11,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-13 21:58:11,561 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676325482126, procId: 270 completed 2023-02-13 21:58:11,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.HMaster$5(2215): Client=jenkins//148.251.75.209 delete testCorruptedSnapshot-1676325482126 2023-02-13 21:58:11,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] procedure2.ProcedureExecutor(1029): Stored pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testCorruptedSnapshot-1676325482126 2023-02-13 21:58:11,567 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676325482126 2023-02-13 21:58:11,568 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=283, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676325482126 2023-02-13 21:58:11,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-02-13 21:58:11,578 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78 2023-02-13 21:58:11,578 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9 2023-02-13 21:58:11,578 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f 2023-02-13 21:58:11,578 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292 2023-02-13 21:58:11,578 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40 2023-02-13 21:58:11,578 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487 2023-02-13 21:58:11,584 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78/cf, FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78/recovered.edits] 2023-02-13 21:58:11,584 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/cf, FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/recovered.edits] 2023-02-13 21:58:11,585 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40/cf, FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40/recovered.edits] 2023-02-13 21:58:11,585 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9/cf, FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9/recovered.edits] 2023-02-13 21:58:11,585 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487/cf, FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487/recovered.edits] 2023-02-13 21:58:11,585 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292/cf, FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292/recovered.edits] 2023-02-13 21:58:11,588 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9/cf/aa2607acaf274ac8a033abba623fad77 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9/cf/aa2607acaf274ac8a033abba623fad77 2023-02-13 21:58:11,589 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487/cf/a015f28c44be4a2f913a58fe70ca099e to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487/cf/a015f28c44be4a2f913a58fe70ca099e 2023-02-13 21:58:11,589 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/cf/263d4d83a7ee44d9a8f0df3d680818c6 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/cf/263d4d83a7ee44d9a8f0df3d680818c6 2023-02-13 21:58:11,589 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78/cf/b4c0b7d99e734eb1bb2d11d74845fa68 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78/cf/b4c0b7d99e734eb1bb2d11d74845fa68 2023-02-13 21:58:11,589 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40/cf/26b9577a905343918c9c51119195cf7c to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40/cf/26b9577a905343918c9c51119195cf7c 2023-02-13 21:58:11,590 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292/cf/6dc812778aed4605b9ff3224823d6d0f to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292/cf/6dc812778aed4605b9ff3224823d6d0f 2023-02-13 21:58:11,590 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9/cf/b25fb604e9074f249714afdbfac4c6d3 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9/cf/b25fb604e9074f249714afdbfac4c6d3 2023-02-13 21:58:11,590 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487/cf/c70f1e666c604cf797b385efc5d7af97 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487/cf/c70f1e666c604cf797b385efc5d7af97 2023-02-13 21:58:11,591 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78/cf/f68183f886e84213becb66fd697bf854 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78/cf/f68183f886e84213becb66fd697bf854 2023-02-13 21:58:11,591 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40/cf/278f4c4405a04414a1eb5319d07c2091 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40/cf/278f4c4405a04414a1eb5319d07c2091 2023-02-13 21:58:11,591 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292/cf/8e37f961530e4488a37efc85ab1c86ac to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292/cf/8e37f961530e4488a37efc85ab1c86ac 2023-02-13 21:58:11,592 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/recovered.edits/18.seqid to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f/recovered.edits/18.seqid 2023-02-13 21:58:11,593 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/00d462c2fbf63cefa67cf8e8f681246f 2023-02-13 21:58:11,593 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9/recovered.edits/18.seqid to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9/recovered.edits/18.seqid 2023-02-13 21:58:11,593 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/f244b6e82e7a1752e9a3cb5f28dea2e9 2023-02-13 21:58:11,593 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487/recovered.edits/18.seqid to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487/recovered.edits/18.seqid 2023-02-13 21:58:11,594 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78/recovered.edits/18.seqid to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78/recovered.edits/18.seqid 2023-02-13 21:58:11,594 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/ac1d4ae96948f4de74aa3114b7308487 2023-02-13 21:58:11,594 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40/recovered.edits/18.seqid to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40/recovered.edits/18.seqid 2023-02-13 21:58:11,594 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/1c996d2a582a7ec43382d5a39bb54c78 2023-02-13 21:58:11,594 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/d1acdd49740dd716d5bec22786474a40 2023-02-13 21:58:11,595 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292/recovered.edits/18.seqid to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292/recovered.edits/18.seqid 2023-02-13 21:58:11,595 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/.tmp/data/default/testCorruptedSnapshot-1676325482126/ef75de132134b515dbe11abe569ac292 2023-02-13 21:58:11,595 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1676325482126 regions 2023-02-13 21:58:11,595 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a 2023-02-13 21:58:11,596 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf] 2023-02-13 21:58:11,598 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/a87ff679a2f3e71d9181a67b7542122c202302130484dfc9fc0b4b8790133632a55a501e to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/a87ff679a2f3e71d9181a67b7542122c202302130484dfc9fc0b4b8790133632a55a501e 2023-02-13 21:58:11,598 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/a87ff679a2f3e71d9181a67b7542122c20230213ba864453080947f097171ffb4373beab to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/a87ff679a2f3e71d9181a67b7542122c20230213ba864453080947f097171ffb4373beab 2023-02-13 21:58:11,599 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/c4ca4238a0b923820dcc509a6f75849b20230213120a326c5afb489fa15b503a4197b66f to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/c4ca4238a0b923820dcc509a6f75849b20230213120a326c5afb489fa15b503a4197b66f 2023-02-13 21:58:11,600 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/c4ca4238a0b923820dcc509a6f75849b2023021391c649b08d6f4b43b12afda4c89899e7 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/c4ca4238a0b923820dcc509a6f75849b2023021391c649b08d6f4b43b12afda4c89899e7 2023-02-13 21:58:11,601 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/c81e728d9d4c2f636f067f89cc14862c202302135f82ed0773c04f148549aaf42ad53f50 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/c81e728d9d4c2f636f067f89cc14862c202302135f82ed0773c04f148549aaf42ad53f50 2023-02-13 21:58:11,603 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/c81e728d9d4c2f636f067f89cc14862c20230213caf8e27114ce40648900d149e1450fb6 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/c81e728d9d4c2f636f067f89cc14862c20230213caf8e27114ce40648900d149e1450fb6 2023-02-13 21:58:11,604 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/d41d8cd98f00b204e9800998ecf8427e20230213481947820bd64f349ecb71771348d25d to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/d41d8cd98f00b204e9800998ecf8427e20230213481947820bd64f349ecb71771348d25d 2023-02-13 21:58:11,604 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/d41d8cd98f00b204e9800998ecf8427e202302136ca58bd8a658451cb2d779c5510856a9 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/d41d8cd98f00b204e9800998ecf8427e202302136ca58bd8a658451cb2d779c5510856a9 2023-02-13 21:58:11,605 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/e4da3b7fbbce2345d7772b0674a318d5202302135a2601c9b72e49aaa036c357b67807bf to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/e4da3b7fbbce2345d7772b0674a318d5202302135a2601c9b72e49aaa036c357b67807bf 2023-02-13 21:58:11,606 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/e4da3b7fbbce2345d7772b0674a318d520230213afc9688d0a4e4dbfbacc2eb854736faa to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/e4da3b7fbbce2345d7772b0674a318d520230213afc9688d0a4e4dbfbacc2eb854736faa 2023-02-13 21:58:11,607 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213455659ab23484199a9c928783d9387f6 to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213455659ab23484199a9c928783d9387f6 2023-02-13 21:58:11,608 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213683c128b9eaf468a8ade77032e7ee80a to hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/archive/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213683c128b9eaf468a8ade77032e7ee80a 2023-02-13 21:58:11,609 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/mobdir/data/default/testCorruptedSnapshot-1676325482126/202bf9f4954e6505eb58f19a6413558a 2023-02-13 21:58:11,610 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=283, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676325482126 2023-02-13 21:58:11,613 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testCorruptedSnapshot-1676325482126 from hbase:meta 2023-02-13 21:58:11,614 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'testCorruptedSnapshot-1676325482126' descriptor. 2023-02-13 21:58:11,615 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=283, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676325482126 2023-02-13 21:58:11,615 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'testCorruptedSnapshot-1676325482126' from region states. 2023-02-13 21:58:11,616 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325491615"}]},"ts":"9223372036854775807"} 2023-02-13 21:58:11,616 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325491615"}]},"ts":"9223372036854775807"} 2023-02-13 21:58:11,616 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325491615"}]},"ts":"9223372036854775807"} 2023-02-13 21:58:11,616 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325491615"}]},"ts":"9223372036854775807"} 2023-02-13 21:58:11,616 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325491615"}]},"ts":"9223372036854775807"} 2023-02-13 21:58:11,616 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676325491615"}]},"ts":"9223372036854775807"} 2023-02-13 21:58:11,617 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-13 21:58:11,617 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 1c996d2a582a7ec43382d5a39bb54c78, NAME => 'testCorruptedSnapshot-1676325482126,,1676325482128.1c996d2a582a7ec43382d5a39bb54c78.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => f244b6e82e7a1752e9a3cb5f28dea2e9, NAME => 'testCorruptedSnapshot-1676325482126,1,1676325482128.f244b6e82e7a1752e9a3cb5f28dea2e9.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => ac1d4ae96948f4de74aa3114b7308487, NAME => 'testCorruptedSnapshot-1676325482126,2,1676325482128.ac1d4ae96948f4de74aa3114b7308487.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => d1acdd49740dd716d5bec22786474a40, NAME => 'testCorruptedSnapshot-1676325482126,3,1676325482128.d1acdd49740dd716d5bec22786474a40.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => ef75de132134b515dbe11abe569ac292, NAME => 'testCorruptedSnapshot-1676325482126,4,1676325482128.ef75de132134b515dbe11abe569ac292.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 00d462c2fbf63cefa67cf8e8f681246f, NAME => 'testCorruptedSnapshot-1676325482126,5,1676325482128.00d462c2fbf63cefa67cf8e8f681246f.', STARTKEY => '5', ENDKEY => ''}] 2023-02-13 21:58:11,617 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'testCorruptedSnapshot-1676325482126' as deleted. 2023-02-13 21:58:11,617 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676325482126","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676325491617"}]},"ts":"9223372036854775807"} 2023-02-13 21:58:11,619 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table testCorruptedSnapshot-1676325482126 state from META 2023-02-13 21:58:11,620 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=283, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676325482126 2023-02-13 21:58:11,621 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=283, state=SUCCESS; DeleteTableProcedure table=testCorruptedSnapshot-1676325482126 in 57 msec 2023-02-13 21:58:11,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-02-13 21:58:11,820 INFO [Listener at localhost.localdomain/44075] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testCorruptedSnapshot-1676325482126, procId: 283 completed 2023-02-13 21:58:11,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(748): Client=jenkins//148.251.75.209 delete name: "emptySnaptb-1676325482126" 2023-02-13 21:58:11,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1676325482126 2023-02-13 21:58:11,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] master.MasterRpcServices(748): Client=jenkins//148.251.75.209 delete name: "snaptb0-1676325482126" 2023-02-13 21:58:11,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46557] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1676325482126 2023-02-13 21:58:11,853 INFO [Listener at localhost.localdomain/44075] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=416 (was 405) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1115766016_15 at /127.0.0.1:40878 [Waiting for operation #272] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x43e34ef3-shared-pool-35 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_682603622_15 at /127.0.0.1:58714 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x43e34ef3-shared-pool-34 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1023233772_15 at /127.0.0.1:40868 [Waiting for operation #262] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1023233772_15 at /127.0.0.1:34140 [Waiting for operation #769] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x43e34ef3-shared-pool-36 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1023233772_15 at /127.0.0.1:41462 [Waiting for operation #185] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1115766016_15 at /127.0.0.1:40852 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: hconnection-0x43e34ef3-shared-pool-33 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_682603622_15 at /127.0.0.1:46652 [Waiting for operation #18] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1023233772_15 at /127.0.0.1:34716 [Waiting for operation #666] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1d9ca414-shared-pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_682603622_15 at /127.0.0.1:34718 [Waiting for operation #179] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1115766016_15 at /127.0.0.1:34128 [Waiting for operation #760] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1115766016_15 at /127.0.0.1:40644 [Waiting for operation #13] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=786 (was 766) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=409 (was 292) - SystemLoadAverage LEAK? -, ProcessCount=164 (was 164), AvailableMemoryMB=6394 (was 6741) 2023-02-13 21:58:11,855 INFO [Listener at localhost.localdomain/44075] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-13 21:58:11,855 INFO [Listener at localhost.localdomain/44075] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-02-13 21:58:11,855 DEBUG [Listener at localhost.localdomain/44075] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0ad20338 to 127.0.0.1:58580 2023-02-13 21:58:11,856 DEBUG [Listener at localhost.localdomain/44075] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 21:58:11,856 DEBUG [Listener at localhost.localdomain/44075] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-13 21:58:11,856 DEBUG [Listener at localhost.localdomain/44075] util.JVMClusterUtil(257): Found active master hash=1497629117, stopped=false 2023-02-13 21:58:11,856 INFO [Listener at localhost.localdomain/44075] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase20.apache.org,46557,1676325450344 2023-02-13 21:58:11,858 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 21:58:11,858 INFO [Listener at localhost.localdomain/44075] procedure2.ProcedureExecutor(629): Stopping 2023-02-13 21:58:11,858 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 21:58:11,859 DEBUG [Listener at localhost.localdomain/44075] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7bbaed0e to 127.0.0.1:58580 2023-02-13 21:58:11,859 DEBUG [Listener at localhost.localdomain/44075] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 21:58:11,859 INFO [Listener at localhost.localdomain/44075] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,36339,1676325451407' ***** 2023-02-13 21:58:11,859 INFO [Listener at localhost.localdomain/44075] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 21:58:11,859 INFO [Listener at localhost.localdomain/44075] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,45015,1676325451448' ***** 2023-02-13 21:58:11,859 INFO [Listener at localhost.localdomain/44075] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 21:58:11,859 INFO [Listener at localhost.localdomain/44075] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,39523,1676325451480' ***** 2023-02-13 21:58:11,859 INFO [Listener at localhost.localdomain/44075] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 21:58:11,859 INFO [RS:2;jenkins-hbase20:39523] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 21:58:11,859 INFO [RS:1;jenkins-hbase20:45015] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 21:58:11,860 INFO [RS:2;jenkins-hbase20:39523] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 21:58:11,860 INFO [RS:1;jenkins-hbase20:45015] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 21:58:11,860 INFO [RS:1;jenkins-hbase20:45015] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 21:58:11,860 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 21:58:11,860 INFO [RS:1;jenkins-hbase20:45015] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:58:11,860 DEBUG [RS:1;jenkins-hbase20:45015] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x422bdf92 to 127.0.0.1:58580 2023-02-13 21:58:11,860 DEBUG [RS:1;jenkins-hbase20:45015] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 21:58:11,860 INFO [RS:1;jenkins-hbase20:45015] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,45015,1676325451448; all regions closed. 2023-02-13 21:58:11,860 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 21:58:11,859 INFO [RS:0;jenkins-hbase20:36339] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 21:58:11,861 INFO [RS:0;jenkins-hbase20:36339] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 21:58:11,861 INFO [RS:0;jenkins-hbase20:36339] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 21:58:11,861 INFO [RS:0;jenkins-hbase20:36339] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:58:11,861 DEBUG [RS:0;jenkins-hbase20:36339] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4413fc87 to 127.0.0.1:58580 2023-02-13 21:58:11,861 DEBUG [RS:0;jenkins-hbase20:36339] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 21:58:11,861 INFO [RS:0;jenkins-hbase20:36339] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 21:58:11,861 INFO [RS:0;jenkins-hbase20:36339] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 21:58:11,861 INFO [RS:0;jenkins-hbase20:36339] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 21:58:11,861 INFO [RS:0;jenkins-hbase20:36339] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-13 21:58:11,860 INFO [RS:2;jenkins-hbase20:39523] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 21:58:11,861 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 21:58:11,862 INFO [RS:2;jenkins-hbase20:39523] regionserver.HRegionServer(3304): Received CLOSE for 5a6c89db834bbded2000d426b959f122 2023-02-13 21:58:11,862 INFO [RS:2;jenkins-hbase20:39523] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:58:11,862 DEBUG [RS:2;jenkins-hbase20:39523] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2825b05c to 127.0.0.1:58580 2023-02-13 21:58:11,862 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 5a6c89db834bbded2000d426b959f122, disabling compactions & flushes 2023-02-13 21:58:11,862 DEBUG [RS:2;jenkins-hbase20:39523] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 21:58:11,862 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676325453392.5a6c89db834bbded2000d426b959f122. 2023-02-13 21:58:11,863 INFO [RS:2;jenkins-hbase20:39523] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-02-13 21:58:11,863 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676325453392.5a6c89db834bbded2000d426b959f122. 2023-02-13 21:58:11,863 INFO [RS:0;jenkins-hbase20:36339] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-02-13 21:58:11,863 DEBUG [RS:0;jenkins-hbase20:36339] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-02-13 21:58:11,863 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 21:58:11,863 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 21:58:11,863 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 21:58:11,863 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 21:58:11,863 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 21:58:11,863 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=137.88 KB heapSize=212.75 KB 2023-02-13 21:58:11,863 DEBUG [RS:2;jenkins-hbase20:39523] regionserver.HRegionServer(1479): Online Regions={5a6c89db834bbded2000d426b959f122=hbase:namespace,,1676325453392.5a6c89db834bbded2000d426b959f122.} 2023-02-13 21:58:11,863 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676325453392.5a6c89db834bbded2000d426b959f122. after waiting 0 ms 2023-02-13 21:58:11,864 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676325453392.5a6c89db834bbded2000d426b959f122. 2023-02-13 21:58:11,864 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing 5a6c89db834bbded2000d426b959f122 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-13 21:58:11,866 DEBUG [RS:0;jenkins-hbase20:36339] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-02-13 21:58:11,867 DEBUG [RS:2;jenkins-hbase20:39523] regionserver.HRegionServer(1505): Waiting on 5a6c89db834bbded2000d426b959f122 2023-02-13 21:58:11,889 DEBUG [RS:1;jenkins-hbase20:45015] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/oldWALs 2023-02-13 21:58:11,889 INFO [RS:1;jenkins-hbase20:45015] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C45015%2C1676325451448:(num 1676325452932) 2023-02-13 21:58:11,889 DEBUG [RS:1;jenkins-hbase20:45015] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 21:58:11,889 INFO [RS:1;jenkins-hbase20:45015] regionserver.LeaseManager(133): Closed leases 2023-02-13 21:58:11,897 INFO [RS:1;jenkins-hbase20:45015] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-13 21:58:11,898 INFO [RS:1;jenkins-hbase20:45015] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 21:58:11,898 INFO [RS:1;jenkins-hbase20:45015] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 21:58:11,898 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 21:58:11,898 INFO [RS:1;jenkins-hbase20:45015] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 21:58:11,904 INFO [RS:1;jenkins-hbase20:45015] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:45015 2023-02-13 21:58:11,907 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=131.12 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/meta/1588230740/.tmp/info/cf1f0155e85b4f4caacab139a30f368c 2023-02-13 21:58:11,913 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for cf1f0155e85b4f4caacab139a30f368c 2023-02-13 21:58:11,919 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 21:58:11,928 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 21:58:11,929 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 21:58:11,930 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.01 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/meta/1588230740/.tmp/rep_barrier/9f2899348eb14c94b1276d8dc5507218 2023-02-13 21:58:11,932 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9f2899348eb14c94b1276d8dc5507218 2023-02-13 21:58:11,944 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.75 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/meta/1588230740/.tmp/table/74aef28b2409411396c9e71be7e733a3 2023-02-13 21:58:11,946 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 74aef28b2409411396c9e71be7e733a3 2023-02-13 21:58:11,947 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/meta/1588230740/.tmp/info/cf1f0155e85b4f4caacab139a30f368c as hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/meta/1588230740/info/cf1f0155e85b4f4caacab139a30f368c 2023-02-13 21:58:11,950 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for cf1f0155e85b4f4caacab139a30f368c 2023-02-13 21:58:11,950 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/meta/1588230740/info/cf1f0155e85b4f4caacab139a30f368c, entries=64, sequenceid=307, filesize=11.9 K 2023-02-13 21:58:11,951 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/meta/1588230740/.tmp/rep_barrier/9f2899348eb14c94b1276d8dc5507218 as hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/meta/1588230740/rep_barrier/9f2899348eb14c94b1276d8dc5507218 2023-02-13 21:58:11,953 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9f2899348eb14c94b1276d8dc5507218 2023-02-13 21:58:11,953 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/meta/1588230740/rep_barrier/9f2899348eb14c94b1276d8dc5507218, entries=18, sequenceid=307, filesize=7.0 K 2023-02-13 21:58:11,954 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/meta/1588230740/.tmp/table/74aef28b2409411396c9e71be7e733a3 as hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/meta/1588230740/table/74aef28b2409411396c9e71be7e733a3 2023-02-13 21:58:11,956 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 74aef28b2409411396c9e71be7e733a3 2023-02-13 21:58:11,957 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/meta/1588230740/table/74aef28b2409411396c9e71be7e733a3, entries=22, sequenceid=307, filesize=7.0 K 2023-02-13 21:58:11,957 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~137.88 KB/141192, heapSize ~212.70 KB/217808, currentSize=0 B/0 for 1588230740 in 94ms, sequenceid=307, compaction requested=false 2023-02-13 21:58:11,959 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): regionserver:39523-0x1015dd830e00003, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 21:58:11,959 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): regionserver:45015-0x1015dd830e00002, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 21:58:11,961 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): regionserver:36339-0x1015dd830e00001, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 21:58:11,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 21:58:11,964 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): regionserver:39523-0x1015dd830e00003, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:58:11,964 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): regionserver:39523-0x1015dd830e00003, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 21:58:11,964 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): regionserver:45015-0x1015dd830e00002, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:58:11,964 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): regionserver:45015-0x1015dd830e00002, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 21:58:11,964 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 21:58:11,964 ERROR [Listener at localhost.localdomain/44075-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@74c235da rejected from java.util.concurrent.ThreadPoolExecutor@70fdbf9a[Shutting down, pool size = 1, active threads = 1, queued tasks = 1, completed tasks = 3] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 21:58:11,964 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): regionserver:36339-0x1015dd830e00001, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,45015,1676325451448 2023-02-13 21:58:11,965 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): regionserver:36339-0x1015dd830e00001, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 21:58:11,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45015-0x1015dd830e00002, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 21:58:11,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39523-0x1015dd830e00003, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 21:58:11,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36339-0x1015dd830e00001, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 21:58:11,968 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/meta/1588230740/recovered.edits/310.seqid, newMaxSeqId=310, maxSeqId=1 2023-02-13 21:58:11,968 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-13 21:58:11,970 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 21:58:11,970 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676325491863Running coprocessor pre-close hooks at 1676325491863Disabling compacts and flushes for region at 1676325491863Disabling writes for close at 1676325491863Obtaining lock to block concurrent updates at 1676325491863Preparing flush snapshotting stores in 1588230740 at 1676325491863Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=141192, getHeapSize=217808, getOffHeapSize=0, getCellsCount=997 at 1676325491864 (+1 ms)Flushing stores of hbase:meta,,1.1588230740 at 1676325491865 (+1 ms)Flushing 1588230740/info: creating writer at 1676325491865Flushing 1588230740/info: appending metadata at 1676325491876 (+11 ms)Flushing 1588230740/info: closing flushed file at 1676325491876Flushing 1588230740/rep_barrier: creating writer at 1676325491913 (+37 ms)Flushing 1588230740/rep_barrier: appending metadata at 1676325491915 (+2 ms)Flushing 1588230740/rep_barrier: closing flushed file at 1676325491916 (+1 ms)Flushing 1588230740/table: creating writer at 1676325491932 (+16 ms)Flushing 1588230740/table: appending metadata at 1676325491934 (+2 ms)Flushing 1588230740/table: closing flushed file at 1676325491934Flushing 1588230740/info: reopening flushed file at 1676325491948 (+14 ms)Flushing 1588230740/rep_barrier: reopening flushed file at 1676325491951 (+3 ms)Flushing 1588230740/table: reopening flushed file at 1676325491954 (+3 ms)Finished flush of dataSize ~137.88 KB/141192, heapSize ~212.70 KB/217808, currentSize=0 B/0 for 1588230740 in 94ms, sequenceid=307, compaction requested=false at 1676325491957 (+3 ms)Writing region close event to WAL at 1676325491964 (+7 ms)Running coprocessor post-close hooks at 1676325491968 (+4 ms)Closed at 1676325491970 (+2 ms) 2023-02-13 21:58:11,970 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-13 21:58:12,066 INFO [RS:0;jenkins-hbase20:36339] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,36339,1676325451407; all regions closed. 2023-02-13 21:58:12,067 DEBUG [RS:2;jenkins-hbase20:39523] regionserver.HRegionServer(1505): Waiting on 5a6c89db834bbded2000d426b959f122 2023-02-13 21:58:12,086 DEBUG [RS:0;jenkins-hbase20:36339] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/oldWALs 2023-02-13 21:58:12,086 INFO [RS:0;jenkins-hbase20:36339] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C36339%2C1676325451407.meta:.meta(num 1676325453081) 2023-02-13 21:58:12,100 DEBUG [RS:0;jenkins-hbase20:36339] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/oldWALs 2023-02-13 21:58:12,100 INFO [RS:0;jenkins-hbase20:36339] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C36339%2C1676325451407:(num 1676325452932) 2023-02-13 21:58:12,100 DEBUG [RS:0;jenkins-hbase20:36339] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 21:58:12,100 INFO [RS:0;jenkins-hbase20:36339] regionserver.LeaseManager(133): Closed leases 2023-02-13 21:58:12,100 INFO [RS:0;jenkins-hbase20:36339] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-13 21:58:12,101 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 21:58:12,102 INFO [RS:0;jenkins-hbase20:36339] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:36339 2023-02-13 21:58:12,267 DEBUG [RS:2;jenkins-hbase20:39523] regionserver.HRegionServer(1505): Waiting on 5a6c89db834bbded2000d426b959f122 2023-02-13 21:58:12,267 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,45015,1676325451448] 2023-02-13 21:58:12,269 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,45015,1676325451448; numProcessing=1 2023-02-13 21:58:12,269 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): regionserver:39523-0x1015dd830e00003, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:58:12,269 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): regionserver:36339-0x1015dd830e00001, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,36339,1676325451407 2023-02-13 21:58:12,269 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 21:58:12,269 ERROR [Listener at localhost.localdomain/44075-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@231e8c73 rejected from java.util.concurrent.ThreadPoolExecutor@27210cc8[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 21:58:12,308 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/namespace/5a6c89db834bbded2000d426b959f122/.tmp/info/4b3598e1ee6744fc9bb5670c4437fd80 2023-02-13 21:58:12,312 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/namespace/5a6c89db834bbded2000d426b959f122/.tmp/info/4b3598e1ee6744fc9bb5670c4437fd80 as hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/namespace/5a6c89db834bbded2000d426b959f122/info/4b3598e1ee6744fc9bb5670c4437fd80 2023-02-13 21:58:12,315 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/namespace/5a6c89db834bbded2000d426b959f122/info/4b3598e1ee6744fc9bb5670c4437fd80, entries=2, sequenceid=6, filesize=4.8 K 2023-02-13 21:58:12,316 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 5a6c89db834bbded2000d426b959f122 in 452ms, sequenceid=6, compaction requested=false 2023-02-13 21:58:12,320 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/data/hbase/namespace/5a6c89db834bbded2000d426b959f122/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-13 21:58:12,322 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676325453392.5a6c89db834bbded2000d426b959f122. 2023-02-13 21:58:12,322 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 5a6c89db834bbded2000d426b959f122: Waiting for close lock at 1676325491862Running coprocessor pre-close hooks at 1676325491862Disabling compacts and flushes for region at 1676325491862Disabling writes for close at 1676325491864 (+2 ms)Obtaining lock to block concurrent updates at 1676325491864Preparing flush snapshotting stores in 5a6c89db834bbded2000d426b959f122 at 1676325491864Finished memstore snapshotting hbase:namespace,,1676325453392.5a6c89db834bbded2000d426b959f122., syncing WAL and waiting on mvcc, flushsize=dataSize=78, getHeapSize=472, getOffHeapSize=0, getCellsCount=2 at 1676325491865 (+1 ms)Flushing stores of hbase:namespace,,1676325453392.5a6c89db834bbded2000d426b959f122. at 1676325491866 (+1 ms)Flushing 5a6c89db834bbded2000d426b959f122/info: creating writer at 1676325491867 (+1 ms)Flushing 5a6c89db834bbded2000d426b959f122/info: appending metadata at 1676325491871 (+4 ms)Flushing 5a6c89db834bbded2000d426b959f122/info: closing flushed file at 1676325491872 (+1 ms)Flushing 5a6c89db834bbded2000d426b959f122/info: reopening flushed file at 1676325492312 (+440 ms)Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 5a6c89db834bbded2000d426b959f122 in 452ms, sequenceid=6, compaction requested=false at 1676325492316 (+4 ms)Writing region close event to WAL at 1676325492317 (+1 ms)Running coprocessor post-close hooks at 1676325492320 (+3 ms)Closed at 1676325492322 (+2 ms) 2023-02-13 21:58:12,322 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676325453392.5a6c89db834bbded2000d426b959f122. 2023-02-13 21:58:12,367 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): regionserver:45015-0x1015dd830e00002, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 21:58:12,367 INFO [RS:1;jenkins-hbase20:45015] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,45015,1676325451448; zookeeper connection closed. 2023-02-13 21:58:12,367 ERROR [Listener at localhost.localdomain/44075-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@61891f30 rejected from java.util.concurrent.ThreadPoolExecutor@70fdbf9a[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 21:58:12,368 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): regionserver:45015-0x1015dd830e00002, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 21:58:12,368 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@55dbaf24] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@55dbaf24 2023-02-13 21:58:12,368 ERROR [Listener at localhost.localdomain/44075-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@191cf45b rejected from java.util.concurrent.ThreadPoolExecutor@70fdbf9a[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 21:58:12,368 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,45015,1676325451448 already deleted, retry=false 2023-02-13 21:58:12,368 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase20.apache.org,45015,1676325451448 expired; onlineServers=2 2023-02-13 21:58:12,369 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,36339,1676325451407] 2023-02-13 21:58:12,369 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,36339,1676325451407; numProcessing=2 2023-02-13 21:58:12,370 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,36339,1676325451407 already deleted, retry=false 2023-02-13 21:58:12,370 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase20.apache.org,36339,1676325451407 expired; onlineServers=1 2023-02-13 21:58:12,468 INFO [RS:2;jenkins-hbase20:39523] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,39523,1676325451480; all regions closed. 2023-02-13 21:58:12,468 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): regionserver:36339-0x1015dd830e00001, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 21:58:12,468 INFO [RS:0;jenkins-hbase20:36339] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,36339,1676325451407; zookeeper connection closed. 2023-02-13 21:58:12,469 ERROR [Listener at localhost.localdomain/44075-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4a1c913 rejected from java.util.concurrent.ThreadPoolExecutor@27210cc8[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 21:58:12,469 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@fadc49d] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@fadc49d 2023-02-13 21:58:12,469 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): regionserver:36339-0x1015dd830e00001, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 21:58:12,469 ERROR [Listener at localhost.localdomain/44075-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7c63b105 rejected from java.util.concurrent.ThreadPoolExecutor@27210cc8[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 21:58:12,474 DEBUG [RS:2;jenkins-hbase20:39523] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/oldWALs 2023-02-13 21:58:12,474 INFO [RS:2;jenkins-hbase20:39523] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C39523%2C1676325451480:(num 1676325452932) 2023-02-13 21:58:12,474 DEBUG [RS:2;jenkins-hbase20:39523] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 21:58:12,474 INFO [RS:2;jenkins-hbase20:39523] regionserver.LeaseManager(133): Closed leases 2023-02-13 21:58:12,474 INFO [RS:2;jenkins-hbase20:39523] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-13 21:58:12,474 INFO [RS:2;jenkins-hbase20:39523] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 21:58:12,475 INFO [RS:2;jenkins-hbase20:39523] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 21:58:12,475 INFO [RS:2;jenkins-hbase20:39523] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 21:58:12,475 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 21:58:12,476 INFO [RS:2;jenkins-hbase20:39523] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:39523 2023-02-13 21:58:12,477 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): regionserver:39523-0x1015dd830e00003, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,39523,1676325451480 2023-02-13 21:58:12,477 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 21:58:12,478 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,39523,1676325451480] 2023-02-13 21:58:12,478 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,39523,1676325451480; numProcessing=3 2023-02-13 21:58:12,479 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,39523,1676325451480 already deleted, retry=false 2023-02-13 21:58:12,479 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase20.apache.org,39523,1676325451480 expired; onlineServers=0 2023-02-13 21:58:12,479 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,46557,1676325450344' ***** 2023-02-13 21:58:12,479 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-02-13 21:58:12,480 DEBUG [M:0;jenkins-hbase20:46557] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@18790b86, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-02-13 21:58:12,480 INFO [M:0;jenkins-hbase20:46557] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,46557,1676325450344 2023-02-13 21:58:12,480 INFO [M:0;jenkins-hbase20:46557] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,46557,1676325450344; all regions closed. 2023-02-13 21:58:12,480 DEBUG [M:0;jenkins-hbase20:46557] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 21:58:12,480 DEBUG [M:0;jenkins-hbase20:46557] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-13 21:58:12,480 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-13 21:58:12,480 DEBUG [M:0;jenkins-hbase20:46557] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-13 21:58:12,480 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1676325452569] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1676325452569,5,FailOnTimeoutGroup] 2023-02-13 21:58:12,480 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1676325452567] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1676325452567,5,FailOnTimeoutGroup] 2023-02-13 21:58:12,480 INFO [M:0;jenkins-hbase20:46557] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-13 21:58:12,481 INFO [M:0;jenkins-hbase20:46557] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-13 21:58:12,481 INFO [M:0;jenkins-hbase20:46557] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-02-13 21:58:12,481 DEBUG [M:0;jenkins-hbase20:46557] master.HMaster(1502): Stopping service threads 2023-02-13 21:58:12,481 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 21:58:12,481 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 21:58:12,481 INFO [M:0;jenkins-hbase20:46557] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-13 21:58:12,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 21:58:12,481 ERROR [M:0;jenkins-hbase20:46557] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[HFileArchiver-3,5,PEWorkerGroup] Thread[HFileArchiver-4,5,PEWorkerGroup] Thread[HFileArchiver-5,5,PEWorkerGroup] Thread[HFileArchiver-6,5,PEWorkerGroup] Thread[HFileArchiver-7,5,PEWorkerGroup] Thread[HFileArchiver-8,5,PEWorkerGroup] 2023-02-13 21:58:12,482 INFO [M:0;jenkins-hbase20:46557] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-13 21:58:12,482 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-13 21:58:12,482 DEBUG [M:0;jenkins-hbase20:46557] zookeeper.ZKUtil(398): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-13 21:58:12,482 WARN [M:0;jenkins-hbase20:46557] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-13 21:58:12,482 INFO [M:0;jenkins-hbase20:46557] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-13 21:58:12,482 INFO [M:0;jenkins-hbase20:46557] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-13 21:58:12,483 DEBUG [M:0;jenkins-hbase20:46557] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 21:58:12,483 INFO [M:0;jenkins-hbase20:46557] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 21:58:12,483 DEBUG [M:0;jenkins-hbase20:46557] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 21:58:12,483 DEBUG [M:0;jenkins-hbase20:46557] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 21:58:12,483 DEBUG [M:0;jenkins-hbase20:46557] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 21:58:12,483 INFO [M:0;jenkins-hbase20:46557] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=844.78 KB heapSize=1.01 MB 2023-02-13 21:58:12,496 INFO [M:0;jenkins-hbase20:46557] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=844.78 KB at sequenceid=2061 (bloomFilter=true), to=hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/997b417a60e84850b4c324d0cd5e1f77 2023-02-13 21:58:12,499 INFO [M:0;jenkins-hbase20:46557] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 997b417a60e84850b4c324d0cd5e1f77 2023-02-13 21:58:12,500 DEBUG [M:0;jenkins-hbase20:46557] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/997b417a60e84850b4c324d0cd5e1f77 as hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/997b417a60e84850b4c324d0cd5e1f77 2023-02-13 21:58:12,503 INFO [M:0;jenkins-hbase20:46557] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 997b417a60e84850b4c324d0cd5e1f77 2023-02-13 21:58:12,503 INFO [M:0;jenkins-hbase20:46557] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37775/user/jenkins/test-data/b620894a-8499-5dc4-ae0e-adda9b5e2750/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/997b417a60e84850b4c324d0cd5e1f77, entries=282, sequenceid=2061, filesize=30.6 K 2023-02-13 21:58:12,503 INFO [M:0;jenkins-hbase20:46557] regionserver.HRegion(2947): Finished flush of dataSize ~844.78 KB/865051, heapSize ~1.01 MB/1057728, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 20ms, sequenceid=2061, compaction requested=false 2023-02-13 21:58:12,504 INFO [M:0;jenkins-hbase20:46557] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 21:58:12,504 DEBUG [M:0;jenkins-hbase20:46557] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676325492483Disabling compacts and flushes for region at 1676325492483Disabling writes for close at 1676325492483Obtaining lock to block concurrent updates at 1676325492483Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676325492483Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=865051, getHeapSize=1057728, getOffHeapSize=0, getCellsCount=2548 at 1676325492483Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676325492483Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676325492483Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676325492487 (+4 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676325492487Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676325492500 (+13 ms)Finished flush of dataSize ~844.78 KB/865051, heapSize ~1.01 MB/1057728, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 20ms, sequenceid=2061, compaction requested=false at 1676325492504 (+4 ms)Writing region close event to WAL at 1676325492504Closed at 1676325492504 2023-02-13 21:58:12,507 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 21:58:12,507 INFO [M:0;jenkins-hbase20:46557] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-13 21:58:12,508 INFO [M:0;jenkins-hbase20:46557] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:46557 2023-02-13 21:58:12,515 DEBUG [M:0;jenkins-hbase20:46557] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,46557,1676325450344 already deleted, retry=false 2023-02-13 21:58:12,578 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): regionserver:39523-0x1015dd830e00003, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 21:58:12,578 INFO [RS:2;jenkins-hbase20:39523] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,39523,1676325451480; zookeeper connection closed. 2023-02-13 21:58:12,579 ERROR [Listener at localhost.localdomain/44075-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@618879b5 rejected from java.util.concurrent.ThreadPoolExecutor@65b7e15c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 8] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 21:58:12,579 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): regionserver:39523-0x1015dd830e00003, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 21:58:12,579 ERROR [Listener at localhost.localdomain/44075-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@33b8935e rejected from java.util.concurrent.ThreadPoolExecutor@65b7e15c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 8] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 21:58:12,579 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@584f7d59] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@584f7d59 2023-02-13 21:58:12,579 INFO [Listener at localhost.localdomain/44075] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-02-13 21:58:12,679 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 21:58:12,679 INFO [M:0;jenkins-hbase20:46557] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,46557,1676325450344; zookeeper connection closed. 2023-02-13 21:58:12,679 ERROR [Listener at localhost.localdomain/44075-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2cffdedf rejected from java.util.concurrent.ThreadPoolExecutor@20d5ac19[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 28] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 21:58:12,679 DEBUG [Listener at localhost.localdomain/44075-EventThread] zookeeper.ZKWatcher(600): master:46557-0x1015dd830e00000, quorum=127.0.0.1:58580, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 21:58:12,679 ERROR [Listener at localhost.localdomain/44075-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@64b200d0 rejected from java.util.concurrent.ThreadPoolExecutor@20d5ac19[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 28] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 21:58:12,682 WARN [Listener at localhost.localdomain/44075] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 21:58:12,686 INFO [Listener at localhost.localdomain/44075] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 21:58:12,791 WARN [BP-658449826-148.251.75.209-1676325447478 heartbeating to localhost.localdomain/127.0.0.1:37775] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 21:58:12,791 WARN [BP-658449826-148.251.75.209-1676325447478 heartbeating to localhost.localdomain/127.0.0.1:37775] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-658449826-148.251.75.209-1676325447478 (Datanode Uuid 26a0f3f3-fab9-4883-9582-fbed1359d57e) service to localhost.localdomain/127.0.0.1:37775 2023-02-13 21:58:12,793 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/cluster_c8eab63c-2879-73ac-9f0f-8254d034d5a9/dfs/data/data5/current/BP-658449826-148.251.75.209-1676325447478] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 21:58:12,794 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/cluster_c8eab63c-2879-73ac-9f0f-8254d034d5a9/dfs/data/data6/current/BP-658449826-148.251.75.209-1676325447478] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 21:58:12,795 WARN [Listener at localhost.localdomain/44075] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 21:58:12,798 INFO [Listener at localhost.localdomain/44075] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 21:58:12,901 WARN [BP-658449826-148.251.75.209-1676325447478 heartbeating to localhost.localdomain/127.0.0.1:37775] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 21:58:12,901 WARN [BP-658449826-148.251.75.209-1676325447478 heartbeating to localhost.localdomain/127.0.0.1:37775] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-658449826-148.251.75.209-1676325447478 (Datanode Uuid 20ff377b-2559-4f6e-af23-080f5371a3dc) service to localhost.localdomain/127.0.0.1:37775 2023-02-13 21:58:12,902 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/cluster_c8eab63c-2879-73ac-9f0f-8254d034d5a9/dfs/data/data3/current/BP-658449826-148.251.75.209-1676325447478] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 21:58:12,903 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/cluster_c8eab63c-2879-73ac-9f0f-8254d034d5a9/dfs/data/data4/current/BP-658449826-148.251.75.209-1676325447478] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 21:58:12,903 WARN [Listener at localhost.localdomain/44075] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 21:58:12,905 INFO [Listener at localhost.localdomain/44075] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 21:58:12,919 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 21:58:13,008 WARN [BP-658449826-148.251.75.209-1676325447478 heartbeating to localhost.localdomain/127.0.0.1:37775] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 21:58:13,008 WARN [BP-658449826-148.251.75.209-1676325447478 heartbeating to localhost.localdomain/127.0.0.1:37775] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-658449826-148.251.75.209-1676325447478 (Datanode Uuid 0a762196-68ef-4608-a8ad-5faf41155fc5) service to localhost.localdomain/127.0.0.1:37775 2023-02-13 21:58:13,009 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/cluster_c8eab63c-2879-73ac-9f0f-8254d034d5a9/dfs/data/data1/current/BP-658449826-148.251.75.209-1676325447478] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 21:58:13,009 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c874f1b1-35ca-6ed7-7719-c2eedf9ff822/cluster_c8eab63c-2879-73ac-9f0f-8254d034d5a9/dfs/data/data2/current/BP-658449826-148.251.75.209-1676325447478] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 21:58:13,061 INFO [Listener at localhost.localdomain/44075] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-02-13 21:58:13,184 INFO [Listener at localhost.localdomain/44075] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-13 21:58:13,253 INFO [Listener at localhost.localdomain/44075] hbase.HBaseTestingUtility(1293): Minicluster is down